Techniques for visualizing storage cluster system configurations and events

ABSTRACT

Various embodiments are generally directed to techniques for generating effective visualizations of some or all of a storage cluster system. An apparatus includes a processor component; a rendering component to generate a visualization of at least a portion of a storage cluster system for presentation on a display, the visualization to comprise a depiction of an object that corresponds to a component of the storage cluster system; and an interpretation component to interpret received indications of operation of an input device to select the depicted object and to select a first time and a second time along a timeline presented on the display, and to generate a command to request information indicating a change in state of the object between the first and second times.

BACKGROUND

Storage cluster systems are often assembled from a complex collection ofhardware and software components that may be selected from a wide rangeof options. Such selections are often based on storage requirements thatdiffer from client to client. Also, as time passes, client needs maychange and available options for replacing and/or upgrading hardware orsoftware components of a storage cluster system may change. Thus, eachstorage cluster system may be of a relatively unique configuration fromthe date of its installation and/or may become relatively unique overtime. This may make lessons learned during the operation of one storagecluster system harder to apply to another.

Additionally, larger storage cluster systems may be made up ofsignificant greater quantities of components and/or those components maybe more thoroughly geographically dispersed. Such measures may be toincrease overall storage capacity, to increase speed of access to clientdata and/or to increase redundancy to more ably handle a wider varietyof failures that may occur. Also, those components may also beinterconnected by a larger quantity and/or greater complexity of networkconnections.

As a result, administrators of storage cluster systems may encountervarious challenges in diagnosing problems, maintaining, repairing and/orupgrading such storage cluster systems. By way of example,administrators charged with overseeing multiple storage cluster systemsmay find it difficult to grasp the details of enough of the componentsof a storage cluster system to understand interactions among thosecomponents, in addition to grasping the details of an event and/orvarious automated responses to that event.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example embodiment of an administration systemexchanging data with multiple storage cluster systems.

FIG. 2A illustrates an example embodiment of an administration system.

FIG. 2B illustrates an alternate example embodiment of an administrationsystem.

FIG. 3 illustrates an example embodiment of a storage cluster system.

FIG. 4A illustrates an example embodiment of a pair of high availabilitygroups of a cluster.

FIG. 4B illustrates an example embodiment of a pair of high availabilitygroups of different clusters.

FIG. 5 illustrates an example embodiment of a HA group of partnerednodes.

FIG. 6 illustrates an example embodiment of duplication and storage ofmetadata within a shared set of storage devices.

FIG. 7 illustrates an example embodiment of a mesh of communicationssessions among nodes.

FIG. 8A illustrates an example embodiment of an ingest server of anadministration system.

FIG. 8B illustrates an example embodiment of a database server of anadministration system.

FIG. 8C illustrates an example embodiment of a visualization server ofan administration system.

FIG. 8D illustrates an example embodiment of an administration device ofan administration system.

FIG. 9 illustrates an example embodiment of representing components of astorage cluster system as objects.

FIG. 10A illustrates an example embodiment of requesting an initialvisualization of a selected storage cluster system.

FIG. 10B illustrates an example embodiment of requesting a visualizationdepicting differences in states of storage cluster system components ofFIG. 10A between two time points.

FIG. 10C illustrates an alternate example embodiment of requesting avisualization depicting differences in states of storage cluster systemcomponents of FIG. 10A between two time points.

FIG. 10D illustrates an example embodiment of a visualization depictingdifferences in states of storage cluster system components of FIG. 10Abetween two time points.

FIG. 11A illustrates an example embodiment of using the visualization ofFIG. 10D to request information concerning a component of a storagecluster system.

FIG. 11B illustrates an example embodiment of limiting details depictedin the visualization of FIG. 10D.

FIG. 11C illustrates an alternate example embodiment of limiting detailsdepicted in the visualization of FIG. 10D.

FIG. 11D illustrates an example embodiment of requesting a moretemporally limited visualization based on the visualization of FIG. 10D.

FIG. 12 illustrates a processing architecture according to anembodiment.

DETAILED DESCRIPTION

Various embodiments are generally directed to techniques for generatingeffective visualizations of some or all of a storage cluster system. Aningest server of an administration system may collect informationconcerning various aspects of the operation of one or more storagecluster systems, and may store such information in an account databasethat associates particular storage cluster systems with particularoperators of storage cluster systems. A database server of theadministration system may recurringly access the account database torecurringly generate and/or update system entries for individual storagecluster systems in a system database organized in a manner in whichcomponents of clusters are treated as objects and relationships betweensuch components are treated as properties of those objects. Avisualization server may, through a network, make the system databaseaccessible to one or more administration devices operated byadministrative personnel tasked with overseeing the operation of one ormore storage cluster systems. The visualization server may provide anapplication programming interface (API) for use by the one or moreadministration devices to retrieve information from the information fromthe system database needed to enable the one or more administrationdevices to generate object-based visualizations of at least portions ofstorage cluster systems. Alternatively or additionally, thevisualization server may itself generate such visualizations andtransmit representations thereof to the one or more administrationdevices.

The information collected from each storage cluster system by the ingestserver may include indications of what hardware and/or softwarecomponents make up a storage cluster system, the manner in which thosecomponents are coupled, features of those components that are usedand/or are not used, client applications for which client data isstored, the types and quantities of client data stored, occurrences ofvarious events affecting storage of the client data and their outcomes,and/or the manner in which to contact one or more administrators. Suchevents may include component failures, instances of limits in capacityor bandwidth being reached or exceeded, changes in the configuration ofstorage volumes, installations or changes in components, changes in themanner in which components are coupled, etc. The ingest server may pollone or more storage cluster systems for such information on a recurringbasis, and/or await transmission of such information to the ingestserver by one or more of the storage cluster systems via a network. Oneor more of the storage cluster systems may transmit such information tothe ingest server in response to the occurrence of one or moreparticular events as part of providing a record thereof to theadministration system for subsequent diagnostics.

The storage cluster systems from which the ingest server receives suchinformation may vary greatly in complexity and capability. By way ofexample, one or more of the storage cluster systems may incorporate asingle node providing a single controller of a relatively small quantityof storage devices that may or may not be operated together as an arrayof storage devices. Such relatively simple storage cluster systems mayincorporate relatively few hardware components and/or softwarecomponents, and may simply be used as archival or “backup” storage forthe client data stored within the client devices to guard against lossof client data should a malfunction of one of the client devices occur.As a result, the information transmitted by such a relatively simplestorage cluster system to the ingest server may correspondingly berelatively simple in content. Also such information may be transmittedrelatively infrequently or in response to a change in the componentsand/or configuration of the storage cluster system.

Alternatively and also by way of example, one or more of the storagecluster systems may incorporate multiple nodes and/or numerous storagedevices. Multiple sets of the storage devices may be operated togetheras fault-tolerant arrays on which client data may be stored in afault-tolerant manner that prevents loss of client data in the event ofa malfunction of one of the storage devices. Also, two or more of themultiple nodes may be interconnected to form high-availability (HA)groups of nodes to support redundancy among the controllers provided byeach of the nodes in which one node may take over for the other in theevent of a failure of a node. Further, the multiple nodes and multiplestorage devices may be divided into multiple clusters that may beinstalled at geographically distant locations, but that may beinterconnected in a manner in which the state of the client data storedwithin the storage devices of one cluster may be mirrored in the stateof the client data stored within the storage devices of another cluster.As a result, the information transmitted by such a relatively complexstorage cluster system to the ingest server may correspondingly berelatively complex in content. Also such information may be transmittedrelatively frequently on a timed basis and/or in response to changes inthe components and/or configuration of the storage cluster system, aswell as in response to various events such as a takeover between nodesor other automated resolution to a detected problem.

In various embodiments, the operator of the administration system may bea purveyor of the storage cluster systems from which the ingest serverreceives such information, such as a manufacturer, distributor,reseller, installer and/or repairer of those storage cluster systems.Thus, each of the operators of one or more of those storage clustersystems may be a customer of such a purveyor, and so each of theoperators of one or more of those storage cluster systems may be deemedan account of the operator of the administration system. Each of thosestorage cluster system operators may be a corporate, governmental,non-profit or other entity that employs one or more of such storagecluster systems for use in storing their own data. Alternatively oradditionally, each of those storage cluster system operators may be acorporate, governmental, non-profit or other entity that operates one ormore of such storage cluster systems to provide storage services and/orother services that include storage services to a multitude of end usersof those services. As part of operating one or more of such storagecluster systems, each of those storage cluster system operators mayemploy or otherwise engage the services of one or more administrators tooversee the operation thereof. Those administrators may be responsiblefor allocating available storage resources, maintenance, security,performing upgrade and/or diagnosing failures. Also, the operator of theadministration system may similarly employ or otherwise engage theservices of one or more assisting administrators to assist theadministrators associated with the storage cluster system operators.Indeed, each of such assisting administrators may be assigned aparticular subset of the storage cluster system operators to which theyare to provide such assistance.

Thus, the ingest server may organize the account database to include aseparate account entry for each operator of one or more storage clustersystems from which the ingest server receives information. For eachstorage cluster system operator that operates more than one storagecluster system, the ingest server may further organize each of theirassociated account entries into multiple system entries that eachcorrespond to one of their storage cluster systems. As information isreceived from each of the storage cluster systems, that information maybe stored within the account entry and/or a separate system entryassociated with the operator of that storage cluster system and/orassociated with that one of multiple storage cluster systems operated bythat operator. Alternatively or additionally, the ingest server mayorganize the information received from each of the storage clustersystems by time, such as and not limited to, the time at which an eventoccurred, the time at which a change in the configuration of a storagecluster system was made, and/or the time at which a piece of informationwas received from a storage cluster system.

The database server may access the data stored within each of theaccount entries and/or system entries of the account database on arecurring basis to identify information signaling a change inconfiguration of a storage cluster system and/or an event occurringwithin a storage cluster system triggering an update in a system entryfor that storage cluster system in the system database. The manner inwhich information is organized within the system database and/or withineach of the system entries thereof may be based on treating each storagecluster system and/or each of various components of a storage clustersystem as an object. Thus, at least a portion of each system entry maybe organized to store information about the storage cluster system towhich it corresponds as an object having various properties. Someobjects may correspond to components that are incorporated into othercomponents such that there is a relationship of one or more componentsbeing “inside” another component. Other objects may correspond tohardware components that are electrically coupled or may correspond tosoftware components that communicate with each other. Indications of thekinds of relationships that one object has to another may also betreated as properties of those objects. The database server may maintainorganizational data that specifies the manner in which data associatedwith each object is to be organized or formatted, such as a type of datastructure specified to be used in storing indications of the variousproperties of each object.

Thus, the database server may, in accordance with the organizationaldata, allocate space within a system entry of the system database tostore information associated with each node and/or each storage deviceof a storage cluster system as an object. Indications of suchinformation as manufacturer, date of manufacture, model, version,features, which features are enabled, assigned identifiers, etc. may bestored as properties for each such object. Alternatively oradditionally, combinations of components that have been configured tocooperate to act as a cluster, a HA group, a drive array, etc. may betreated as objects, and indications of the manner in which they arecoupled, RAID level, which are active, which are on standby to takeover, etc. may be stored as properties for each such object. Alsoalternatively or additionally, allocated spaces in which to store datamay be treated as objects such that an aggregate defined within thestorage space provided by one or more storage devices, and which may bedivided into one or more volumes, may be treated as an object and/oreach of those volumes may be treated as an object such that space may beallocated for each within a system entry of the system database.

In some embodiments, indications of the manner in which each object isrelated to one or more other objects may be stored as part of theproperties of that object. In other embodiments, information concerningeach relationship between two or more objects may be separately storedalong with indications of properties of each of those relationships.Thus, an indication of the fact that a particular storage device iscoupled to a particular node or an indication that a particular volumeis defined within a particular aggregate may be stored as part of theproperties of those objects or may be separately stored. Further, amongthe indications of properties stored for each object and/or for eachrelationship among two or more objects may be various temporalindications, such as when a particular object or relationship was addedand/or removed, and/or when various events affecting each object orrelationship occurred. Alternatively or additionally, in variousembodiments, indications of events associated with an object or arelationship among objects may be stored among the properties of anobject or a relationship, or may be separately stored in a datastructure of events that is associated by the system database with anobject or a relationship.

The visualization server may be coupled by one or more networks toadministration devices operated by administrators of one or moreoperators of one or more storage cluster systems, and/or operated byassistant administrators of a purveyor of the one or more storagecluster systems. Through an exchange of commands and informationrequested from a system entry of the system database, the visualizationserver and an administration device may cooperate to provide anadministrator with a visualization of the status of various portions ofa storage cluster system at one or more particular times and/or througha period of time to enhance the ability of the administrator to diagnosecauses of malfunctions and/or to perform other oversight duties.

Initially, as a result of an administrator operating an administrationdevice to request an initial visualization of a particular storagecluster system, that administration device may transmit a command to thevisualization server to get a visualization of that storage clustersystem in its current state. In some embodiments, such an initialcommand may be directly typed by the administrator at a command lineprovided by a user interface (UI) of the administration device. In otherembodiments, the UI may present a menu or other form of listing ofstorage cluster systems from which the administrator may select theparticular storage cluster system, thereby triggering the transmissionof such a command. Regardless of the manner in which the command isgenerated and/or caused to be transmitted to the visualization server,the visualization server may retrieve information from the databaseserver concerning the particular storage cluster system and may relay itto the administration device to be presented on a display as an initialvisualization of the particular storage cluster system to theadministrator.

As previously discussed, information concerning storage cluster systemsis stored within the system database in an object-based manner in whichcomponents and/or subparts of components of the particular storagecluster system may be stored in a manner that treats each as an objectwith associated indications of properties. Thus, the informationretrieved and relayed to the administration device by the visualizationserver may be have such an object-based organization. Further, in someembodiments, the visualization server may provide the object-basedinformation to the administration device and allow the administrationdevice to derive the initial visualization of the particular storagecluster system that it presents on a display to the administrator.However, in other embodiments, the visualization server may, itself,derive and generate the initial visualization, and may transmit arepresentation of the initial visualization to the administration devicethat enables the administration device to simply present the initialvisualization on a display thereof.

Regardless of the manner in which the administration device is caused toand/or enabled to present the initial visualization, the administratoroperating the administration device may make use of the UI provided bythe administration device to interact with the initial visualization inany of a variety of ways to provide an indication of what portion of theparticular storage cluster system the administrator seeks to focus on.In some embodiments, the UI may enable the administrator to select thatportion by any of a variety of highlighting techniques. Alternatively oradditionally, the administrator may make use of the UI to indicate thatthe administrator seeks to view the state of the particular storagecluster system (or the portion thereof) at a time different than thecurrent time. As a result of such operation of the UI, theadministration device may transmit one or more new commands to thevisualization server for different information concerning the particularstorage cluster system needed to enable the presentation of thevisualization requested by the administrator.

As an alternative to requesting a visualization of the state of theparticular storage cluster system (or a portion thereof) at a particulartime point (whether it is the current time or an earlier time), the UImay enable the administrator to specify a range of time and request avisualization of changes occurring in the state of the particularstorage cluster (or a portion thereof) that occurred during that rangeof time. More specifically, the UI may enable the administrator tospecify two time points in a timeline and request a visualization of thedifferences in the state of the particular storage cluster system (or aportion thereof) between those two time points in that timeline. Suchoperation of the UI may trigger the administration device to transmit acommand to the visualization server for information concerning thedifferences between the states of the particular storage cluster systemat those two time points.

In some embodiments, a visualization may include graphical indicationsof how a depicted state of the depicted storage cluster system ordepicted portion thereof compares to one or more best practices. Stateddifferently, a visualization of some or all of a storage cluster systemmay be accompanied by indicators of warnings of one or more risksincurred by one or more aspects of a depicted state thereof and/orsuggested actions to take to at least reduce such risks. In someembodiments, the timeline may include graphical indications of eventsoccurring within a range of times depicted within at least a portion ofthe timeline. Such events may include instances of failures and/orrecoveries therefrom, instances of additions and/or removals ofcomponents, instances of starting and/or stopping of operation of one ormore components, etc. In some embodiments, a visualization depictingdifferences in at least a portion of a storage cluster system betweentwo time points may include graphical indications of particular eventsthat changed a state of at least one component and/or a relationshipbetween at least two components that occurred during the period of timebetween those two time points. Various ones of these indications ofrisks, suggestions and/or events may be color-coded and/or depicted withvarious differing symbols.

With general reference to notations and nomenclature used herein,portions of the detailed description which follows may be presented interms of program procedures executed on a computer or network ofcomputers. These procedural descriptions and representations are used bythose skilled in the art to most effectively convey the substance oftheir work to others skilled in the art. A procedure is here, andgenerally, conceived to be a self-consistent sequence of operationsleading to a desired result. These operations are those requiringphysical manipulations of physical quantities. Usually, though notnecessarily, these quantities take the form of electrical, magnetic oroptical signals capable of being stored, transferred, combined,compared, and otherwise manipulated. It proves convenient at times,principally for reasons of common usage, to refer to these signals asbits, values, elements, symbols, characters, terms, numbers, or thelike. It should be noted, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to those quantities.

Further, these manipulations are often referred to in terms, such asadding or comparing, which are commonly associated with mentaloperations performed by a human operator. However, no such capability ofa human operator is necessary, or desirable in most cases, in any of theoperations described herein that form part of one or more embodiments.Rather, these operations are machine operations. Useful machines forperforming operations of various embodiments include general purposedigital computers as selectively activated or configured by a computerprogram stored within that is written in accordance with the teachingsherein, and/or include apparatus specially constructed for the requiredpurpose. Various embodiments also relate to apparatus or systems forperforming these operations. These apparatus may be speciallyconstructed for the required purpose or may include a general purposecomputer. The required structure for a variety of these machines willappear from the description given.

Reference is now made to the drawings, wherein like reference numeralsare used to refer to like elements throughout. In the followingdescription, for purposes of explanation, numerous specific details areset forth in order to provide a thorough understanding thereof. It maybe evident, however, that the novel embodiments can be practiced withoutthese specific details. In other instances, well known structures anddevices are shown in block diagram form in order to facilitate adescription thereof. The intention is to cover all modifications,equivalents, and alternatives within the scope of the claims.

FIG. 1 illustrates a block diagram of an example embodiment of anadministration system 2000 interacting with multiple storage clustersystems 1000 via a network 999. As depicted, the administration system2000 may incorporate one or more administration devices 2200, an ingestserver 2400, a database server 2500 and/or a visualization server 2600.As also depicted, each of the storage cluster systems 1000 mayincorporate one or more client devices 100, one or more administrationdevices 200, one or more nodes 300 and/or one or more storage devices800. As further depicted, and as will be discussed in greater detail,one or more of the devices of each of the storage cluster systems 1000may exchange data with one or more of the devices of the administrationsystem 2000 via the network 999. The network 999 may be a single networklimited to extending within a single building or other relativelylimited area, may include a combination of connected networks extendinga considerable distance, and/or may include the Internet.

Within each of the storage cluster systems 1000, the one or more nodes300 may control the one or more storage devices 800 to store client datareceived from the one or more client devices. The one or moreadministration devices 200 and/or 2200 may be operated byadministrator(s) to configure aspects of the operation of the one ormore nodes 300 and/or to configure aspects of the manner in which theclient data is stored within the one or more storage devices 800. On arecurring basis, at least one of the nodes 300 of each of the storagecluster systems 1000 may transmit various pieces of informationconcerning the configuration and operation of that one of the storagecluster systems 1000 to the ingest server 2400 of the administrationsystem 2000. Also, the visualization server 2600 may enableadministrators operating the one or more administration devices 200and/or 2200 to obtain object-based visualizations of at least a portionof the storage cluster system 1000.

Within the administration system 2000, the ingest server 2400 may storeand organize the information received via the network 999 from at leastone node 300 of each of the storage cluster systems 1000 concerningvarious aspects of the operation of the storage cluster systems 1000.The ingest server 2400 may store such information in an account databasethat associates particular storage cluster systems with particularaccounts. The database server 2500 may recurringly access the accountdatabase to recurringly generate and/or update system entries forindividual storage cluster systems in a system database organized in amanner in which at least components of clusters are treated as objects,and in which relationships between components such as interconnectionstherebetween may be treated as properties of those components. Thevisualization server 2600 may, through the network 999, make the systemdatabase accessible to the one or more administration devices 200 and/or2200 to enable the presentation of visualizations of at least a portionof one or more of the storage cluster systems 1000 thereby toadministrators tasked with overseeing the operation thereof.

In various embodiments, the operator of the administration system 2000may be a purveyor of the storage cluster systems 1000 from which theingest server 2400 receives such information, such as a manufacturer,distributor, reseller, installer and/or repairer of those storagecluster systems. Thus, each of the operators of one or more of thosestorage cluster systems 1000 may be a customer of such a purveyor. Eachof those storage cluster system operators may be a corporate,governmental, non-profit or other entity that employs one or more ofsuch storage cluster systems for use in storing their own data.Alternatively or additionally, each of those storage cluster systemoperators may operate one or more of such storage cluster systems toprovide storage services and/or other services that require storage to amultitude of end users of those services. As part of operating one ormore of such storage cluster systems, each of those storage clustersystem operators may employ or otherwise engage the services of the oneor more administrators to oversee the operation thereof throughoperation of the one or more administration devices 200 of each of thestorage cluster systems 1000. Also, the operator of the administrationsystem 2000 may similarly employ or otherwise engage the services of theone or more assisting administrators to assist the administratorsassociated with the storage cluster system operators.

FIGS. 2A and 2B each illustrate a block diagram of the administrationsystem 2000 interacting with a storage cluster system 1000 through thenetwork 999 in greater detail. In FIG. 2A, the administration system2000 may interact with a relatively simple embodiment of the storagecluster system 1000 that incorporates a single node 300 that controlsthe one or more storage devices 800. In FIG. 2B, the administrationsystem 2000 may interact with a relatively complex embodiment of thestorage cluster system 1000 that incorporates multiple ones of the nodes300 and of the storage devices 800 that may be organized into multipleclusters 1300 in which the manner in which client data is stored withinone set of the storage devices 800 is mirrored within another set of thestorage devices 800 at what may be geographically distant locations toincrease fault tolerance.

Referring to both FIGS. 2A and 2B, regardless of the degree ofcomplexity of any of the multiple storage cluster systems 1000 withwhich the administration system 2000 interacts, the information receivedby the ingest server 2400 from one of the nodes 300 of each of thestorage cluster systems 1000 is stored within an account entry 2432associated with an operator of one or more of the storage clustersystems 1000. For each such operator, a separate account entry 2432 isdefined within an account database 2430 maintained by the ingest server2400, and each account entry 2432 may include the information receivedfrom all of the storage cluster systems 1000 operated by a single suchoperator.

The database server 2500 recurringly accesses each of the accountentries 2432 to determine if any new information has been receivedconcerning aspects of operation of any of the one or more storagecluster systems 1000 from which the ingest server 2400 may receiveinformation to determine whether any information stored within thesystem database 2530 is to be updated. As depicted, the system database2530 may be made up of one or more system entries 2531, each of whichmay correspond to a single storage cluster system 1000. As previouslydiscussed, within each of the system entries 2531, informationconcerning various aspects of a storage cluster system 1000 is organizedinto objects having properties. As also previously discussed, thevisualization server 2600 receives and responds to commands from one ormore of the administration devices 200 and/or 2200 for informationconcerning at least a portion of one of the one or more storage clustersystems 1000 for which a system entry 2531 may be present within thesystem database 2530.

FIG. 3 illustrates a block diagram of an example embodiment of thestorage cluster system 1000 incorporating the one or more client devices100, the one or more administration devices 200, and/or the one or moreclusters 1300, such as the depicted clusters 1300 a and 1300 z. Asdepicted, the cluster 1300 a may incorporate one or more of the nodes300, such as the depicted nodes 300 a-d, and one or more of the storagedevices 800, such as the depicted sets of storage devices 800 ab and 800cd. As also depicted, the cluster 1300 z may incorporate more of thenodes 300, such as the depicted nodes 300 y-z, and more of the storagedevices 800, such as the depicted sets of storage devices 800 yz. Asfurther depicted, the cluster 1300 a may include a HA group 1600 abincorporating the nodes 300 a-b as partners and the set of storagedevices 800 ab. The cluster 1300 a may also include a HA group 1600 cdincorporating the nodes 300 c-d as partners and the set of storagedevices 800 cd. Correspondingly, the cluster 1300 z may include a HAgroup 1600 yz incorporating the nodes 300 y-z as partners and the set ofstorage devices 800 yz. It should be noted that within the storagecluster system 1000, each of the clusters 1300 a and 1300 z is aninstance of a cluster 1300, each of the sets of storage devices 800 ab,800 cd and 800 yz represents one or more instances of a storage device800, and each of the nodes 300 a-d and 300 y-z is an instance of thenode 300 as earlier depicted and discussed in reference to FIGS. 1 and2.

In some embodiments, the clusters 1300 a and 1300 z may be positioned atgeographically distant locations to enable a degree of redundancy instoring and retrieving client data 130 provided by one or more of theclient devices 100 for storage. Such positioning may be deemed desirableto enable continued access to the client data 130 by one or more of theclient devices 100 and/or the administration device 200 despite afailure or other event that may render one or the other of the clusters1300 a or 1300 z inaccessible thereto. As depicted, one or both of theclusters 1300 a and 1300 z may additionally store other client data 131that may be entirely unrelated to the client data 130.

The formation of the HA group 1600 ab with at least the two nodes 300 aand 300 b partnered to share access to the set of storage devices 800 abmay enable a degree of fault tolerance in accessing the client data 130as stored within the set of storage devices 800 ab by enabling one ofthe nodes 300 a-b in an inactive state to take over for its partner inan active state (e.g., the other of the nodes 300 a-b) in response to anerror condition within that active one of the nodes 300 a-b.Correspondingly, the formation of the HA group 1600 yz with at least thetwo nodes 300 y and 300 z partnered to share access to the set ofstorage devices 800 yz may similarly enable a degree of fault tolerancein accessing the client data 130 as stored within the set of storagedevices 800 yz by similarly enabling one of the nodes 300 y-z in aninactive state to similarly take over for its partner in active state(e.g., the other of the nodes 300 y-z).

As depicted, any active one of the nodes 300 a-d and 300 y-z may be madeaccessible to the client devices 100 and/or the administration device200 via a client interconnect 199. As also depicted, the nodes 300 a-dand 300 y-z may be additionally coupled via an inter-clusterinterconnect 399. In some embodiments, the interconnects 199 and 399 mayboth extend through the same network 999. Each of the interconnects 199and 399 may be implemented as virtual private networks (VPNs) definedusing any of a variety of network security protocols through the network999. Again, the network 999 may be a single network limited to extendingwithin a single building or other relatively limited area, may include acombination of connected networks extending a considerable distance,and/or may include the Internet. As an alternative to coexisting withinthe same network 999, the interconnects 199 and 399 may be implementedas entirely physically separate networks. By way of example, the clientinterconnect 199 may extend through the Internet to enable the clientdevices 100 and/or the administration device 200 to be positioned atgeographically diverse locations, while the inter-cluster interconnect399 may extend through a leased line between the two geographicallydistant locations at which each of the clusters 1300 a and 1300 z arepositioned.

As depicted, the partnered nodes within each of the HA groups 1600 ab,1600 cd and 1600 yz may be additionally coupled via HA interconnects 699ab, 699 cd and 699 yz, respectively. As also depicted, the nodes withineach of the HA groups 1600 ab, 1600 cd and 1600 yz may be coupled to thesets of storage devices 800 ab, 800 cd and 800 yz in a manner enablingshared access via storage interconnects 899 ab, 899 cd and 899 yz,respectively. The partnered nodes and set of storage devices making upeach of the HA groups 1600 ab, 1600 cd and 1600 yz may be positionedwithin relatively close physical proximity to each other such that theinterconnects 699 ab, 899 ab, 699 cd, 899 cd, 699 yz and 899 yz may eachtraverse a relatively short distance (e.g., extending within a roomand/or within a cabinet).

More broadly, one or more of the interconnects 199, 399, 699 ab, 699 cdand 699 yz may be based on any of a variety (or combination) ofcommunications technologies by which signals may be exchanged, includingwithout limitation, wired technologies employing electrically and/oroptically conductive cabling, and wireless technologies employinginfrared, radio frequency or other forms of wireless transmission. Eachof the interconnects 899 ab, 899 cd and 899 yz may be based on any of avariety of widely known and used storage interface standards, includingand not limited to, SCSI, serially-attached SCSI (SAS), Fibre Channel,etc.

It should be noted that despite the depiction of specific quantities ofclusters and nodes within the storage cluster system 1000, otherembodiments are possible that incorporate different quantities ofclusters and nodes. Similarly, despite the depiction of specificquantities of HA groups and nodes within each of the clusters 1300 a and1300 z, other embodiments are possible that incorporate differingquantities of HA groups and nodes. Further, although each of the HAgroups 1600 ab, 1600 cd and 1600 yz is depicted as incorporating a pairof nodes 300 a-b, 300 c-d and 300 y-z, respectively, other embodimentsare possible in which one or more of the HA groups 1600 ab, 1600 cd and1600 yz may incorporate more than two nodes.

FIGS. 4A and 4B each illustrate a block diagram of an example portion ofthe embodiment of the storage cluster system 1000 of FIG. 3 in greaterdetail. More specifically, FIG. 4A depicts aspects of the nodes 300 a-dand interconnections thereamong within the cluster 1300 a in greaterdetail. FIG. 4B depicts aspects of the interconnections among the nodes300 a-b and 300 y-z, including interconnections extending between theclusters 1300 a and 1300 z, in greater detail.

Referring to both FIGS. 4A and 4B, each of the nodes 300 a-d and 300 y-zmay incorporate one or more of a Managing module 400, a Network module500 and a Data module 600. As depicted, each of the Managing modules 400and the Network modules 500 may be coupled to the client interconnect199, by which each may be accessible to one or more of the clientdevices 100, the administration device 200, and/or to the administrationsystem 2000. The Managing module 400 of one or more active ones of thenodes 300 a-d and 300 y-z may cooperate with the administration device200 via the client interconnect 199 to allow an operator of theadministration device 200 to configure various aspects of the manner inwhich the storage cluster system 1000 stores and provides access to theclient data 130 provided by one or more of the client devices 100. Thatsame Managing module 400 may also recurringly transmit indications ofthat configuration and other information concerning the storage clustersystem 1000 to the ingest server 2400 of the administration system 2000.The Network module 500 of one or more active ones of the nodes 300 a-dand 300 y-z may receive and respond to requests for storage servicesreceived from one or more of the client devices 100 via the clientinterconnect 199, and may perform a protocol conversion to translateeach storage service request into one or more data access commands.

As depicted, the Data modules 600 of all of the nodes 300 a-d and 300y-z may be coupled to each other via the inter-cluster interconnect 399.Also, within each of the HA groups 1600 ab, 1600 cd and 1600 yz, Datamodules 600 of partnered nodes may share couplings to the sets ofstorage devices 800 ab, 800 cd and 800 yz, respectively. Morespecifically, the Data modules 600 of the partnered nodes 300 a and 300b may both be coupled to the set of storage devices 800 ab via thestorage interconnect 899 ab, the Data modules 600 of the partnered nodes300 c and 300 d may both be coupled to the set of storage devices 800 cdvia the storage interconnect 899 cd, and the Data modules 600 of thepartnered nodes 300 y and 300 z may both be coupled to the set ofstorage devices 800 yz via the storage interconnect 899 yz. The Datamodules 600 of active ones of the nodes 300 a-d and 300 y-z may performthe data access commands derived by one or more of the Network modules500 of these nodes from translating storage service requests receivedfrom one or more of the client devices 100.

Thus, the Data modules 600 of active ones of the nodes 300 a-d and 300y-z may access corresponding ones of the sets of storage devices 800 ab,800 cd and 800 yz via corresponding ones of the storage interconnects899 ab, 899 cd and 899 yz to store and/or retrieve client data 130 aspart of performing the data access commands. The data access commandsmay be accompanied by portions of the client data 130 to store and/ornewer portions of the client data 130 with which to update the clientdata 130 as stored. Alternatively or additionally, the data accesscommands may specify portions of the client data 130 to be retrievedfrom storage for provision back to one or more of the client devices100.

Further, and referring to FIG. 4B, the Data module 600 of an active oneof the nodes 300 a-b and 300 y-z of one of the clusters 1300 a or 1300 zmay replicate the data access commands and transmit the resultingreplica data access commands via the inter-cluster interconnect 399 toanother active one of the nodes 300 a-b and 300 y-z of the other of theclusters 1300 a or 1300 z to enable at least partial parallelperformance of the data access commands by two of the Data modules 600.In this way, the state of the client data 130 as stored within one ofthe sets of storage devices 800 ab or 800 yz may be mirrored within theother of the sets of storage devices 800 ab or 800 yz, as depicted.

Such mirroring of the state of the client data 130 between multiple setsof storage devices associated with different clusters that may begeographically distant from each other may be deemed desirable toaddress the possibility of the nodes of one of the clusters becominginaccessible as a result of a regional failure of the clientinterconnect 199 (e.g., as a result of a failure of a portion of thenetwork 999 through which a portion of the client interconnect extendsin a particular geographic region). As familiar to those skilled in theart, the use of additional interconnect(s) between partnered nodes of aHA group (e.g., the HA interconnects 699 ab, 699 cd and 699 yz) tends toencourage physically locating partnered nodes of a HA group in closeproximity to each other such that a localized failure of a network mayrender all nodes of a HA group inaccessible to the client devices 100.For example, a failure of a portion of a network that includes theclient interconnect 199 in the vicinity of both of the nodes 300 a and300 b may render both of the nodes 300 a and 300 b inaccessible to theclient devices 100 such that the client data 130 stored within the setsof storage devices 800 ab becomes inaccessible through either of thenodes 300 a or 300 b. With both of the sets of the storage devices 800ab and 800 yz mirroring the state of the client data 130, the clientdevices 100 are still able to access the client data 130 within the setof storage devices 800 yz, despite the loss of access to the set ofstorage devices 800 ab.

Referring again to both FIGS. 4A and 4B, and as previously discussed,the sharing of access via the storage interconnects 899 ab, 899 cd and899 yz to each of the sets of storage devices 800 ab, 800 cd and 800 yz,respectively, among partnered ones of the nodes 300 a-d and 300 y-z mayenable continued access to one of the sets of storage devices 800 ab,800 cd and 800 yz in the event of a failure occurring within one of thenodes 300 a-d and 300 y-z. The coupling of Data modules 600 of partneredones of the nodes 300 a-d and 300 y-z within each of the HA groups 1600ab, 1600 cd and 1600 yz via the HA interconnects 699 ab, 699 cd and 699yz, respectively, may enable such continued access in spite of such afailure. Through the HA interconnects 699 ab, 699 cd or 699 yz, Datamodules 600 of each of these nodes may each monitor the status of theData modules 600 their partners. More specifically, the Data modules 600of the partnered nodes 300 a and 300 b may monitor each other throughthe HA interconnect 699 ab, the Data modules 600 of the partnered nodes300 c and 300 d may monitor each other through the HA interconnect 699cd, and the Data modules 600 of the partnered nodes 300 y and 300 z maymonitor each other through the HA interconnect 699 yz.

Such monitoring may entail recurring exchanges of “heartbeat” and/orother status signals (e.g., messages conveying the current state ofperformance of a data access command) via one or more of the HAinterconnects 699 ab, 699 cd or 699 yz in which an instance of anabsence of receipt of such a signal within a specified recurringinterval may be taken as an indication of a failure of the one of theData modules 600 from which the signal was expected. Alternatively oradditionally, such monitoring may entail awaiting an indication from amonitored one of the Data modules 600 that a failure of anothercomponent of one of the nodes 300 a-d or 300 y-z has occurred, such as afailure of a Managing module 400 and/or of a Network module 500 of thatone of the nodes 300 a-d or 300 y-z. In response to such an indicationof failure of an active one of the nodes 300 a-d or 300 y-z belonging toone of the HA groups 1600 ab, 1600 cd or 1600 yz, an inactive partneramong the nodes 300 a-d or 300 y-z of the same one of the HA groups 1600ab, 1600 cd or 1600 yz may take over. Such a “takeover” betweenpartnered ones of the nodes 300 a-d or 300 y-z may be a completetakeover inasmuch as the partner that is taking over may take overperformance of all of the functions that were performed by the failingone of these nodes.

However, in some embodiments, at least the Network modules 500 and theData modules 600 of multiple ones of the nodes 300 a-d and/or 300 y-zmay be interconnected in a manner enabling a partial takeover inresponse to the failure of a portion of one of the nodes 300 a-d or 300y-z. Referring more specifically to FIG. 5A, the Network modules 500 ofeach of the nodes 300 a-d may be coupled to the Data modules 600 of eachof the nodes 300 a-d via an intra-cluster interconnect 599 a. In otherwords, within the cluster 1300 a, all of the Network modules 500 and allof the Data modules 600 may be coupled to enable data access commands tobe exchanged between Network modules 500 and Data modules 600 ofdifferent ones of the nodes 300 a-d. Thus, by way of example, where theNetwork module 500 of the node 300 a has failed, but the Data module 600of the node 300 a is still operable, the Network module 500 of itspartner node 300 b (or of one of the nodes 300 c or 300 d with which thenode 300 a is not partnered in a HA group) may take over for the Networkmodule 500 of the node 300 a.

Although the clusters 1300 a and 1300 z may be geographically distantfrom each other, within each of the clusters 1300 a and 1300 z, nodesand/or components of nodes may be positioned within relatively closephysical proximity to each other such that the intra-clusterinterconnects 599 a and 599 z may each traverse a relatively shortdistance (e.g., extending within a room and/or within a single cabinet).More broadly, one or more of the intra-cluster interconnects 599 a and599 z may be based on any of a variety (or combination) ofcommunications technologies by which signals may be exchanged, includingwithout limitation, wired technologies employing electrically and/oroptically conductive cabling, and wireless technologies employinginfrared, radio frequency or other forms of wireless transmission. Byway of example, the intra-cluster interconnect 599 a may be made up of amesh of point-to-point interconnects coupling each Network module 500 ofeach of the nodes 300 a-d to each Data module 600 of each of the nodes300 a-d. Alternatively, by way of another example, the intra-clusterinterconnect 599 a may include a network switch (not shown) to whicheach of the Network modules 500 and each of the Data modules 600 of thenodes 300 a-d may be coupled.

The Managing module 400 of one or more of the active ones of the nodes300 a-d and 300 y-z may recurringly retrieve indications of status fromthe Network modules 500 and/or Data modules 600 within the same nodeand/or from others of the nodes 300 a-d and 300 y-z. Where necessary,such a Managing module 400 may indirectly retrieve such information fromone or more Network modules 500 and/or Data modules 600 through one ormore other Managing modules 400. Among such retrieved indications may beindications of a failure in a Network module 500 and/or a Data module600, and such a failure may have prompted a partial or a completetakeover by one of the nodes 300 a-d and 300 y-z of functions performedby another of the nodes 300 a-d and 300 y-z. Correspondingly, followinga repair or other correction to address such a failure, the retrievedindications may include an indication of a “give-back” event in which apartial or complete takeover is reversed. In some embodiments, aManaging module 400 that recurringly retrieves such indications ofstatus may recurringly transmit those indications to the ingest server2400 of the administration system 2000. Alternatively or additionally,that Managing module 400 may generate a summary or other form ofaggregation of such events as takeovers and give-backs to transmit tothe ingest server 2400.

It should also be noted that despite the depiction of only a single oneof each of the Managing module 400, the Network module 500 and the Datamodule 600 within each of the nodes 300 a-d and 300 y-z, otherembodiments are possible that may incorporate different quantities ofone or more of the Managing module 400, the Network module 500 and theData module 600 within one or more of these nodes. By way of example,embodiments are possible in which one or more of the nodes 300 a-dand/or 300 y-z incorporate more than one Network module 500 to provide adegree of fault-tolerance within a node for communications with one ormore of the client devices 100, and/or incorporate more than one Datamodule 600 to provide a degree of fault-tolerance within a node foraccessing a corresponding one of the sets of storage devices 800 ab, 800cd or 800 yz.

FIG. 5 illustrates a block diagram of an example embodiment of the HAgroup 1600 ab of the cluster 1300 a of the embodiment of the storagecluster system 1000 of FIG. 3 in greater detail. As depicted, of thenodes 300 a and 300 b of the HA group 1600 ab, the node 300 a may beactive to engage in communications with a client device 100 and/or theadministration device 200, and may be active to perform operationsaltering the client data 130 within the set of storage devices 800 ab,while the node 300 b may be inactive and awaiting a need to take overfor the node 300 a. More specifically, the Managing module 400 and theNetwork module 500 of the node 300 a may engage in communications withthe client devices 100, the administration device 200 and/or the ingestserver 2400 of the administration system 2000 (as indicated with theManaging module 400 and the Network module 500 of the node 300 a beingdrawn with solid lines), while the Managing module 400 and the Networkmodule 500 of the node 300 b may not (as indicated with the Managingmodule 400 and the Network module 500 being drawn with dotted lines).

In various embodiments, the Managing module 400 of each of the nodes 300a-b incorporates one or more of a processor component 450, a memory 460and an interface 490 to couple the Managing module 400 to at least theclient interconnect 199. The memory 460 may store a control routine 440.The control routine 440 may incorporate a sequence of instructionsoperative on the processor component 450 in its role as a main processorcomponent of the Managing module 400 to implement logic to performvarious functions. As a result of the node 300 a being active to engagein communications with one or more of the client devices 100 and/or theadministration device 200, the processor component 450 of the Managingmodule 400 of the node 300 a may be active to execute the controlroutine 440. In contrast, as a result of the node 300 b being inactive,the processor component 450 may not be active to execute the controlroutine 440 within the Managing module 400 of the node 300 b. However,if the node 300 b takes over for the node 300 a, then the controlroutine 440 within the node 300 b may begin to be executed, while thecontrol routine 440 within the node 300 a may cease to be executed.

In executing the control routine 440, the processor component 450 of theManaging module 400 of the active node 300 a may operate the interface490 to accept remotely supplied configuration data. In some embodiments,such remote configuration data may emanate from the administrationdevice 200. By way of example, which one(s) of the nodes 300 b-d or 300y-z may be partnered to form one or more HA groups (e.g., the HA groups1600 ab, 1600 cd or 1600 yz) may be remotely configured, as well as whatnodes and/or HA groups may cooperate to provide further fault tolerance(e.g., geographically dispersed fault tolerance), what network addressesmay be allocated to one or more of the nodes 300 a-d and/or 300 y-z onvarious interconnects, etc. In other embodiments, such remoteconfiguration may emanate from one or more of the client devices 100.The processor component 450 may provide a web server, telnet access,instant messaging and/or other communications service(s) by which suchaspects of operation may be remotely configured from the administrationdevice 200 or one or more of the client devices 100 via the clientinterconnect 199. Regardless of the exact manner in which configurationinformation is remotely provided, as the processor component 450receives such configuration information and/or subsequent to receivingsuch information, the processor component 450 may operate the interface490 to relay it and/or updates thereto to the Network module 500 and/orthe Data module 600 as a portion of metadata. Alternatively oradditionally, the processor component 450 may also operate the interface490 to relay such configuration information and/or updates thereto tothe ingest server 2400 of the administration system 2000.

In various embodiments, the Network module 500 of each of the nodes 300a-b incorporates one or more of a processor component 550, a memory 560and an interface 590 to couple the Network module 500 to one or both ofthe client interconnect 199 and the intra-cluster interconnect 599 a.The memory 560 may store a control routine 540. The control routine 540may incorporate a sequence of instructions operative on the processorcomponent 550 in its role as a main processor component of the Networkmodule 500 to implement logic to perform various functions. As a resultof the node 300 a being active to engage in communications with one ormore of the client devices 100 and to perform data access commands, theprocessor component 550 of the Network module 500 of the node 300 a maybe active to execute the control routine 540. In contrast, as a resultof the node 300 b being inactive, the processor component 550 may not beactive to execute the control routine 540 within the N-module of thenode 300 b. However, if the node 300 b takes over for the node 300 a,then the control routine 540 within the node 300 b may begin to beexecuted, while the control routine 540 within the node 300 a may ceaseto be executed.

In executing the control routine 540, the processor component 550 of theNetwork module 500 of the active node 300 a may operate the interface590 to perform various tests to detect other devices with which tocommunicate and/or assign network addresses by which other devices maybe contacted for communication. At least as part of rebooting followingbeing reset or powered on, the processor component 550 may performvarious tests on the client interconnect 199 and/or the intra-clusterinterconnect 599 a to determine addresses and/or communicationsprotocols for communicating with one or more components (e.g., Managingmodules 400, Network modules 500 and/or Data modules 600) of one or moreof the nodes 300 a-d and/or 300 y-z. Alternatively or additionally, inembodiments in which at least a portion of the intra-clusterinterconnect 599 a supports internet protocol (IP) addressing, theprocessor component 550 may function in the role of a dynamic hostcontrol protocol (DHCP) server to assign such addresses. Alsoalternatively or additionally, the processor component 550 may receiveconfiguration information from the Managing module 400 (e.g., a portionof metadata).

In some embodiments, configuration information received from theManaging module 400 may be employed by the processor component 550 inperforming such tests on the client interconnect 199 and/or theintra-cluster interconnect 599 a (e.g., the configuration information soreceived may include a range of IP addresses to be tested). As theprocessor component 550 performs such tests and/or subsequent toperforming such tests, the processor component 550 may operate theinterface 590 to relay indications of the results of those tests and/orupdates thereto to the Data module 600 as a portion of metadata.Further, as the processor component 550 interacts with one or more ofthe client devices 100 and/or other devices, the processor component 550may detect changes in information determined from the performance ofvarious tests, and may operate the interface 590 to provide indicationsof those changes to the Data module 600 as portions of updated metadata.

In some embodiments, as the processor component 550 of each Networkmodule 500 that performs such tests, those processor components 550 mayalso operate their respective interfaces 590 to relay the results ofthose tests and/or updates thereto to the Managing module 400 that is incommunication with the ingest server 2400, either directly thereto, orthrough another intervening Managing module 400. The Managing module 400in communication with the ingest server 2400 may also transmit a copy ofthe portions of metadata as originally generated and as updated by theresults of those tests. Differences in the portions of metadatapreceding and following such updates may provide an indication to bestored by the ingest server 2400 of an attempt to configure the storagecluster system 1000 that is being defeated by a condition affecting aportion of an interconnect and/or another factor.

In further executing the control routine 540, the processor component550 may operate the interface 590 to exchange storage service requests,responses thereto and/or client data 130 with one or more of the clientdevices 100 via the client interconnect 199. The client devices 100 andthe Network module(s) 500 of one or more active ones of the nodes 300a-d and 300 y-z may interact with each other via the client interconnect199 in accordance with a client/server model for the handling of clientdata 130. Stated differently, each of the client devices 100 may issuerequests for storage services related to the storage of client data 130to one or more of the nodes 300 a-d and 300 y-z that are active toengage in communications with the client devices 100. In so doing, theclient devices 100 and the Network module 500 may exchange packets overthe client interconnect 199 in which storage service requests may betransmitted to the Network module 500, responses (e.g., indications ofstatus of handling of the requests) may be transmitted to the clientdevices 100, and client data 130 may be exchanged therebetween. Theexchanged packets may utilize any of a variety of file-based accessprotocols, including and not limited to, Common Internet File System(CIFS) protocol or Network File System (NFS) protocol, over TCP/IP.Alternatively or additionally, the exchanged packets may utilize any ofa variety of block-based access protocols, including and not limited to,Small Computer Systems Interface (SCSI) protocol encapsulated over TCP(iSCSI) and/or SCSI encapsulated over Fibre Channel (FCP).

Also in executing the control routine 540, the processor component 550may operate the interface 590 to exchange commands and/or data,including client data 130, with the Data module 600 via theintra-cluster interconnect 599 a. Such exchanges of commands and/or datamay or may not employ a protocol in which packets are used. In someembodiments, data access commands to effect exchanges of client data 130may be exchanged through the intra-cluster interconnect 599 a in amanner that may be agnostic of any particular file system that may beselected for use in storing the client data 130 within the set ofstorage devices 800 ab. More specifically, the manner in which portionsof client data 130 may be referred to in data access commands to storeand/or retrieve client data 130 may entail identification of file names,identification of block identifiers, etc. in a manner meant to beindependent of a selection of a file system.

Given the possible differences in protocols and/or other aspects ofcommunications, the processor component 550 may be caused to translatebetween protocols employed in communications with one or more of theclient devices 100 via the client interconnect 199 and protocolsemployed in communications with the Data module 600 via theintra-cluster interconnect 599 a. Alternatively or additionally, one ormore of the protocols employed in communications via the clientinterconnect 199 may employ file and/or block identification in a mannerenabling a minimal degree of protocol translation between suchcommunications and communications via the intra-cluster interconnect 599a.

In performing such protocol translations, the processor component 550may be caused to relay a storage service request from one of the clientdevices 100 to the Data module 600 as one or more data access commandsto store and/or retrieve client data 130. More specifically, a requestreceived via the client interconnect 199 for storage services toretrieve client data 130 may be converted into one or more data accesscommands conveyed to the Data module 600 via the intra-clusterinterconnect 599 a to retrieve client data 130 from the set of storagedevices 800 ab and to provide the client data 130 to the Network module500 to be relayed by the Network module 500 back to the requesting oneof the client devices 100. Also, a request received via the clientinterconnect 199 for storage services to store client data 130 may beconverted into one or more data access commands conveyed to the Datamodule 600 via the intra-cluster interconnect 599 a to store the clientdata 130 within the set of storage devices 800 ab.

In various embodiments, the Data module 600 of each of the nodes 300 a-bincorporates one or more of a processor component 650, a memory 660, astorage controller 665 to couple the Data module 600 to the set ofstorage devices 800 ab via the storage interconnect 899 ab, and aninterface 690 to couple the Data module 600 to one or more of theintra-cluster interconnect 599 a, the inter-cluster interconnect 399 andthe HA interconnect 699 ab. The memory 660 stores one or more of acontrol routine 640 and metadata 630 ab. Also, and as will be explainedin greater detail, in the Data module 600 of the node 300 a, a portionof the memory 660 may be allocated to serve as a synchronization cache(sync cache) 639 a, while a portion of the memory 660 may be similarlyallocated to serve as a sync cache 639 b in the D-module of the node 300b. The control routine 640 incorporates a sequence of instructionsoperative on the processor component 650 in its role as a main processorcomponent of the Data module 600 to implement logic to perform variousfunctions. However, as a result of the node 300 a being active to engagein communications with one or more of the client devices 100 and toperform data access commands, a different portion of the control routine640 may be executed by the processor component 650 of the Data module600 of the node 300 a from a portion of the control routine 640 that maybe executed by the processor component 650 of the D-module of the node300 b. As a result, different logic may be implemented by the executionsof different portions of the control routine 640 within each of theseData modules 600.

In executing the control routine 640, the processor component 650 of theData module 600 of the active node 300 a may operate the interface 690to receive portions of metadata and/or updates thereto from the Managingmodule 400 and/or the Network module 500 via the intra-clusterinterconnect 599 a. Regardless of whether aspects of the operation of atleast the node 300 a are remotely configured via the Managing module 400and/or are configured based on the results of tests performed by theNetwork module 500, the processor component 650 may generate themetadata 630 ab from those received metadata portions indicating theresulting configuration of those aspects, and may store the metadata 630ab within the memory 660 for subsequent use by the processor component650. The processor component 650 may repeat the generation of themetadata 630 ab in response to receiving updated portion(s) of metadatafrom the Managing module 400, the Network module 500 and/or otherpossible sources of updated metadata portions, thereby creating anupdated version of the metadata 630 ab which the processor component 650may store within the memory 660 in place of earlier version(s).Following generation of the metadata 630 ab and/or each updated versionthereof, the processor component 650 may store the metadata 630 abwithin the set of storage devices 800 ab for later retrieval during asubsequent rebooting of at least the Data module 600 of the node 300 a.

Also following generation of the metadata 630 ab and/or each updatedversion thereof, the processor component 650 of the Data module 600 ofthe node 300 a may operate the interface 690 to transmit a duplicate ofthe metadata 630 ab to the Data module 600 of the inactive node 300 bvia the HA interconnect 699 ab to enable the node 300 b to more speedilytake over for the active node 300 a in response to a failure within thenode 300 a. In this way, the node 300 b is directly provided with themetadata 630 ab and/or updated versions thereof to provide informationneeded by the node 300 b to more readily take over communications withone or more client devices, take over communications with one or moreothers of the nodes 300 c-d and/or 300 y-z, and/or take over control ofand/or access to the set of storage devices 800 ab.

Still further following generation of the metadata 630 ab and/or eachupdated version thereof, the processor component 650 of the Data module600 of the node 300 a may operate the interface 690 to transmit aportion of the metadata 630 ab to the Data module 600 of an active oneof the nodes 300 y-z of the HA group 1600 yz of the other cluster 1300z. Alternatively or additionally, the processor component 650 of theData module 600 of the node 300 a may operate the interface 690 totransmit metadata portion(s) received from the Managing module 400and/or the Network module 500 of the node 300 a to the active one of thenodes 300 y-z. Such metadata portion(s) may include indications ofaspects of operation of all of the nodes 300 a-b and 300 y-z together instoring and/or providing access to the client data 130, and may beprovided to the active one of the nodes 300 y-z as an input to othermetadata that may be separately generated and/or maintained by the nodes300 y-z.

In some embodiments, as the processor component 650 of at least the Datamodule 600 receives metadata portions (or updates thereto) and generateseach new version of the metadata 630 ab, the processor component 650 mayoperate the interface 690 to relay each new version of the metadata 630ab to the Managing module 400 that is in communication with the ingestserver 2400 of the administration system 2000 through one or moreNetwork modules 500. As previously discussed, the Managing module 400 incommunication with the ingest server 2400 may also transmit a copies ofthe portions of metadata from which the metadata 630 ab is derived, andin so doing, may transmit a copy of the metadata 630 ab with thosemetadata portions.

In further executing the control routine 640, the processor component650 of the Data module 600 of the node 300 a may operate the set ofstorage devices 800 ab through the storage controller 665 to store andretrieve client data 130 in response to data access commands to do soreceived via the intra-cluster interconnect 599 a, as has beendescribed. The processor component 650 may operate the interface 690 toreceive the data access commands from and/or exchange data (includingclient data 130) with the Network module 500 via the intra-clusterinterconnect 599 a. The processor component 650 may be caused to retrythe performance of a data access command to store or retrieve clientdata 130 at least in response to the occurrence of a short term failurein performance (e.g., a failure that is likely to be resolved relativelyquickly). However, if the failure in performance is a longer termfailure (e.g., a failure that cannot be resolved quickly and/or requiresintervention of personnel), then a takeover may occur in which, forexample, the node 300 b becomes the new active node of the HA group 1600ab.

In addition to operating the storage controller 665 to execute dataaccess commands to store client data 130 within the set of storagedevices 800 ab and/or retrieve client data 130 therefrom, the processorcomponent 650 of the Data module 600 of the node 300 a may alsoreplicate the data access commands and operate the interface 690 totransmit the resulting replica data access commands via theinter-cluster interconnect 399 to a Data module 600 of an active one ofthe nodes 300 y-z of the HA group 1600 yz of the other cluster 1300 z.As has been discussed, the transmission of such replica data accesscommands to an active node of another HA group may provide an additionaldegree of fault tolerance in the storage and/or retrieval of client data130 in which the replica data access commands may be performed by anactive node of another cluster at least partly in parallel with theperformance of the original data access command by the node 300 a. Theprocessor component 650 may be caused to retry the transmission of suchreplica data access commands to either the same active one of the nodes300 y-z within the HA group 1600 yz and/or to a different inactive oneof the nodes 300 y-z within the HA group 1600 yz in response toindications of errors in either the receipt or performance of thereplica data access commands. Retrying transmission of replica dataaccess commands to an inactive one of the nodes 300 y-z may cause orarise from a takeover of the active one of the nodes 300 y-z by theinactive one thereof.

In support of such exchanges of replica data access commands andresponses thereto between the Data module 600 of the node 300 a and aData module 600 of an active one of the nodes 300 y-z, the processorcomponent 650 of the Data module 600 of the node 300 a may employinformation included within the metadata 630 ab to form an activecommunications session with the Data module 600 of that other activenode through the inter-cluster interconnect 399. The processor component650 may additionally form an inactive communications session with aD-module of the inactive one of the nodes 300 y-z through theinter-cluster interconnect 399 in preparation for retrying atransmission of a replica data access command to the Data module 600 ofthat inactive node. Further, if the processor component 650 retries thetransmission of a replica data access command to the Data module 600 ofthat inactive one node, then the processor component 650 may act tochange the state of the inactive communications session formed with theData module 600 of that inactive node from inactive to active.

In executing the control routine 640, the processor component 650 of theData module 600 of the inactive node 300 b may operate the interface 690to receive the metadata 630 ab and/or updates thereto from the Datamodule 600 of the node 300 a via the HA interconnect 699 ab. Theprocessor component 650 may then store the received metadata 630 aband/or the received updates thereto within the memory 660 for subsequentuse. Again, provision of the metadata 630 ab and updates theretodirectly to the node 300 b by the node 300 a may be deemed desirable toenable the node 300 b to more quickly take over for the node 300 a(thereby transitioning from being an inactive node of the HA group 1600ab to becoming the active node of the HA group 1600 ab) in response to afailure occurring within the node 300 a. More specifically, with themetadata 630 ab already provided to the Data module 600 of the node 300b, the need for the processor component 650 of the Data module 600 ofthe node 300 b to take additional time to retrieve the metadata 630 abfrom other sources is alleviated. More precisely, the need for theprocessor component to retrieve the metadata 630 ab from the set ofstorage devices 800 ab, or to request portions of metadata from theManaging module 400 and/or the Network module 500 of either of the nodes300 a or 300 b upon taking over for the node 300 a is alleviated.

As depicted, the metadata 630 ab may include immutable metadata 631 aband mutable metadata 632 ab. What pieces of metadata are included ineach of the immutable metadata 631 ab and the mutable metadata 632 abmay be based on the relative frequency with which each piece of metadatais expected to change. By way of example, aspects of the storage ofclient data 130 within the set of storage devices 800 ab, such as aselection of file system, a “level” of redundancy of a Redundant Arrayof Independent Disks (RAID), etc. may be deemed immutable as a result ofbeing deemed less likely to change or likely to change less frequentlythan other metadata. In contrast, a network address of a M-module, aN-module or a D-module of one of the other nodes 300 a-d or 300 y-z withwhich the node 300 a may communicate via one of the interconnects 399,599 a or 699 ab may be deemed mutable as a result of being deemed morelikely to change or likely to change more frequently than othermetadata.

As part of determining whether one of the nodes 300 a or 300 b needs totake over for the other, the processor components 650 of the D-modulesof each of the nodes 300 a and 300 b may cooperate to recurringlyexchange indications of the status of their nodes via the HAinterconnect 699 ab extending therebetween. As previously discussed suchexchanges of status indications may take the form of recurring“heartbeat” signals and/or indications of the current state ofperforming an operation (e.g., a performing a data access command).Again, an indication that a component of one of the nodes 300 a-b hassuffered a malfunction may be the lack of receipt of an expectedheartbeat signal or other status indication by the other of the nodes300 a-b within a specified period of time (e.g., within a recurringinterval of time). Where the Data module 600 of the active node 300 areceives an indication of a failure within the inactive node 300 b, theprocessor component 650 of the Data module 600 of the node 300 a (oranother component of the node 300 a) may refrain from taking action totake over the node 300 b, since the node 300 b is inactive such that thenode 300 b may not be performing a task that requires a takeover of thenode 300 b.

However, where the Data module 600 of the inactive node 300 b receivesan indication of a failure within the active node 300 a, the processorcomponent 650 of the Data module 600 of the inactive node 300 b (oranother component of the inactive node 300 b) may take action to takeover the node 300 a, since the node 300 a is active to engage incommunications with the client devices 100, to perform data accesscommands, and to cooperate with another active node to cause at leastpartial parallel performance of data access commands therebetween. Byway of example, the processor component 650 of the Data module 600 ofthe node 300 b may signal the Network module 500 of the node 300 b totake over communications with one or more of the client devices 100and/or may begin performing the data access commands that were performedby the processor component 650 of the Data module 600 of the node 300 a.In taking over the performance of those data access commands, theprocessor component 650 of the Data module 600 of the node 300 b maytake over access to and control of the set of storage devices 800 ab viathe coupling that the Data modules 600 of both of the nodes 300 a and300 b share to the set of storage devices 800 ab through the storageinterconnect 899 ab.

Where the inactive node 300 b does take over for the active node 300 ain response to a failure occurring within the node 300 a, the active andinactive roles of the nodes 300 a and 300 b may fully reverse, at leastafter the failure within the node 300 a has been corrected. Morespecifically, the Managing module 400 and the Network module 500 of thenode 300 b may become active to engage in communications with the clientdevices 100 and/or the administration device 200 via the clientinterconnect 199 to receive configuration information and storageservice requests, and thereby take over for the Managing module 400 andthe Network module 500 of the node 300 a, while the Managing module 400and the Network module 500 of the node 300 a become inactive. Similarly,the Data module 600 of the node 300 b may become active to perform andreplicate data access commands, and to transmit replica data accesscommands to another active node via the inter-cluster interconnect 399to cause at least partial parallel performance of the data accesscommands, and thereby take over for the Data module 600 of the node 300a, while the Data module 600 of the node 300 a becomes inactive.However, in becoming active, the processor component 650 of the Datamodule 600 of the now inactive node 300 a may cooperate with theprocessor component 650 of the Data module 600 of the node 300 b toreceive new versions of the metadata 630 ab generated within the node300 b and to exchange indications of status with the Data module 600 ofthe node 300 b via the HA interconnect 699 ab to determine if the node300 a should subsequently take over for the now active node 300 b.

The processor components 650 of the Data modules 600 of each of thenodes 300 a and 300 b may designate or otherwise use a portion ofcorresponding ones of the memories 660 as the sync caches 639 a and 639b, respectively, in communications with Data module(s) 600 of others ofthe nodes 300 a-d and/or 300 y-z. More specifically, the processorcomponents 650 of the Data modules 600 of the nodes 300 a and 300 b mayemploy the sync caches 639 a and 639 b, respectively, to buffer versionsof the metadata 630 ab and/or status indications exchanged therebetween.Alternatively or additionally, the processor component 650 of the Datamodule 600 of the node 300 a may maintain and employ the sync cache 639a to buffer replica data access commands transmitted to another activenode of another HA pair of another cluster and/or indications of statusof performance of those replica data access commands received from thatother active node.

As the processor components 550 of Network modules 500 and the processorcomponents 650 of Data modules 600 within active ones of the nodes 300a-d and 300 y-z execute relevant portions of the control routines 540and 640, respectively, to handle requests for storages services receivedfrom one or more of the client devices 100, each of those processorcomponents 550 and 650 may monitor various aspects of the performanceand usage of the storage cluster system 1000. By way of example, each ofsuch processor components 550 may monitor the rates at which requestsfor storage services are received and relayed, the amount of timerequired to do so, the rate of throughput of client data 130 exchangedthrough active ones of the Network modules 500, and any instances inwhich a specified maximum or other high rate of throughput of clientdata 130 is reached or exceeded. Also by way of example, each of suchprocessor components 650 may monitor the quantities of client data 130stored within and/or amounts of storage capacity still available withinassociated ones of the sets of storage devices 800 ab, 800 cd and/or 800yz, data rates at which client data 130 is stored or retrieved, and anyinstances in which an access to one or more storage devices needed to beretried. Such processor components 550 and 650 may operate correspondingones of the interfaces 590 and 690, respectively, to relay suchinformation to the Managing module 400 that is in communication with theingest server 2400, either directly thereto or through anotherintervening Managing module 400. The one of the Managing modules 400 incommunication with the ingest server 2400 may, in turn, relay suchinformation to the ingest server 2400.

FIG. 6 illustrates a block diagram of another example embodiment of theHA group 1600 ab of the cluster 1300 a of the storage cluster system1000 in greater detail. As again depicted, of the nodes 300 a and 300 bof the HA group 1600 ab, the node 300 a may be active to engage incommunications with a client device 100 and/or the administration device200, and/or may be active to perform operations altering the client data130 within the set of storage devices 800 ab, while the node 300 b maybe inactive and awaiting a need to take over for the node 300 a. FIG. 6also depicts various aspects of the generation, duplication and storageof the metadata 630 ab within the set of storage devices 800 abalongside the client data 130 in greater detail.

Each of the sets of storage devices 800 ab, 800 cd and 800 yz may bemade up of storage devices based on any of a variety of storagetechnologies, including and not limited to, ferromagnetic “hard” or“floppy” drives, magneto-optical media drives, optical media drives,non-volatile solid state drives, etc. As depicted, the set of storagedevices 800 ab may include LUs 862 t-v that may be operated together toform an array of storage devices. In some embodiments, the processorcomponent 650 of the Data module 600 of the node 300 a may operate thestorage controller 665 to treat each of the storage devices of the setof storage devices 800 ab as a separate LU and/or may be caused to treata group of those storage devices as a single LU. Multiple LUs may beoperated together via the storage controller 665 to implement a level ofRAID or other form of array that imparts fault tolerance in the storageof data therein. The manner in which LUs are defined among one or morestorage devices of the set of storage devices 800 ab, and/or the mannerin which multiple LUs may be operated together may be specified withinthe metadata 630 ab.

The processor component 650 may be caused to allocate storage space inany of a variety of ways within a single LU and/or within multiple LUsoperated together to form an array. In so doing, the processor component650 may be caused to subdivide storage space in any of a variety of wayswithin a single LU and/or within multiple LUs that are operatedtogether. By way of example, such subdivisions may be effected as partof organizing client data 130 into separate categories based on subject,as part of separating client data 130 into different versions generatedover time, as part of implementing differing access policies todifferent pieces of client data 130, etc. In some embodiments, and asdepicted, the storage space provided by within the LU 862 t or within acombination of the LUs 862 t-v may be designated as an aggregate 872.Further, the aggregate 872 may be subdivided into volumes 873 p-r. Themanner in which aggregates and/or volumes are defined may be selected toconform to the specification(s) of one or more widely known and usedfile systems, including and not limited to, Write Anywhere File Layout(WAFL). The manner in which aggregates and/or volumes within aggregatesare allocated among a single LU or multiple LUs that are operatedtogether may be specified within the metadata 630 ab.

The client data 130 may be stored entirely within one of the volumes 873p-r, or may be distributed among multiple ones of the volumes 873 p-r(as depicted). As also depicted, the metadata 630 ab may also be storedwithin the set of storage devices 800 ab along with client data 130, atleast within the same aggregate 872. In some embodiments, the metadata630 ab may be stored within one or more of the same volumes 873 p-r asclient data 130 (as depicted). In other embodiments, the metadata 630 abmay be stored within one of the volumes 873 p-r that is separate fromone or more others of the volumes 873 p-r within which client data 130may be stored. The manner in which the metadata 630 ab and/or the clientdata 130 are to be organized within aggregates and/or values may bespecified within the metadata 630 ab itself.

As previously discussed, the Managing module 400 of the active node 300a may provide portions of metadata, including updates thereof, to theNetwork module 500 and/or the Data module 600 in response to receivingconfiguration information from one of the client devices 100. Again,such portions of metadata so provided by the Managing module 400 (and/orupdates thereto) may include configuration information received inconfiguration data from the administration device 200 and/or one or moreof the client devices 100. Also, the Network module 500 of the activenode 300 a may provide portions of metadata, including updates thereof,to the Data module 600 that indicate results of various tests performedby the Network module 500. Again, the portions of metadata so providedby the Network module 500 (and/or updates thereto) may includeconfiguration information derived by the Network module 500 through theperformance of various tests. And again, a duplicate of the metadata 630ab may be generated and stored within the sync cache 639 a as a portionof duplication data 636 ab, by which the duplicate of the metadata 630ab may be transmitted via the interface 690 and the HA interconnect 699ab to the Data module 600 of the inactive node 300 b.

As the processor component 650 of the Data module 600 of one or more ofthe active nodes 300 a-d and 300 y-z are caused to create aggregatesand/or volumes in corresponding ones of the sets of storage devices 800ab, 800 cd and 800 yz, those processor components 650 may monitor theprocess of doing so and record various results of those processes, suchas failures in particular storage devices, instances of needing toresize one aggregate or volume to accommodate an expansion of another,and instances of automatically increasing the size of a volume oraggregate as a result of the storage of a larger quantity of client data130 than could be accommodated by the original defined capacity of thatvolume or aggregate. Again, those ones of the processor component 650may operate corresponding ones of the interfaces 690 to relay suchinformation to the one of the Managing modules 400 that is incommunication with the ingest server 2400 to be relayed thereto.

FIG. 7 depicts an example embodiment of a mesh of communicationssessions formed among the nodes 300 a-b and 300 y-z through theinter-cluster interconnect 399 in greater detail. More specifically,through the inter-cluster interconnect 399, each of the nodes 300 a and300 b of the HA group 1600 ab forms a communications session with eachof the nodes 300 y and 300 z of the HA group 1600 yz, thereby formingthe depicted mesh of communications sessions among the nodes 300 a-b and300 y-z. As depicted, of these communications sessions, thecommunications session extending between the nodes 300 a and 300 y maybe an active communications session (as indicated with a solid line),while the others of these communications sessions may be inactivecommunications sessions (as indicated with dotted lines). This reflectsthe fact that the nodes 300 a and 300 y, at least initially, are eachthe active nodes of the HA groups 1600 ab and 1600 yz, respectively,that engage in communications to exchange replica data access commandsand associated data to enable at least partly parallel performance ofdata access commands between the HA groups 1600 ab and 1600 yz.

Thus, during normal operation of the storage cluster system 1000 inwhich the nodes 300 a and 300 y are active nodes and no errors occurwithin either of the nodes 300 a or 300 y, a request for storageservices is received by the node 300 a via the client interconnect 199from one of the client devices 100. Following conversion of the storageservices request into a data access command by the Network module 500 ofthe node 300 a, the Data module 600 of the node 300 a may both beginperformance of the data access command and transmit a replica of thatdata access command to the node 300 y via the active communicationssession formed through inter-cluster interconnect 399 between the nodes300 a and 300 y. The Data module 600 of the node 300 y may then performthe replica data access command at least partly in parallel with theperformance of the data access command by the Data module 600 of thenode 300 a.

In preparation for such a transmission, the Data module 600 of the node300 a may cooperate with the Data module 600 of the node 300 y to formthe depicted active communications session between the nodes 300 a to300 y through an exchange of messages requesting and accepting formationof the active communications session. Following its formation, the Datamodules 600 of the nodes 300 a and 300 y may cooperate to maintain theactive communications session by recurring exchanges of test signals(e.g., test messages) therethrough to monitor the state of the activecommunications session.

In addition to the Data modules 600 of the nodes 300 a and 300 ycooperating to form and maintain the depicted active communicationssession through the inter-cluster interconnect 399 to support suchexchanges of replica data access commands, the Data modules 600 of allof the nodes 300 a-b and 300 y-z may cooperate to form and maintain thedepicted inactive communications sessions through the inter-clusterinterconnect 399 in preparation for handling an error conditionaffecting one of the nodes 300 a or 300 y. More specifically, testsignals (e.g., test messages) may be exchanged through one or more ofthe inactive communications sessions to monitor their state.

In the event of a failure of at least a portion of the node 300 a, thenode 300 b may take over for the node 300 a, and in so doing, may changethe state of the inactive communications session extending between theData modules 600 of the nodes 300 b and 300 y into an activecommunications session. By doing so, the node 300 b becomes able totransmit replica data access commands to the node 300 y in place of thenode 300 a. Correspondingly, in the event of a failure of at least aportion of the node 300 y, the node 300 z may take over for the node 300y, and in so doing, may change the state of the inactive communicationssession extending between the Data modules 600 of the nodes 300 a and300 z into an active communications session. By doing so, the node 300 zbecomes able to receive and perform replica data access commands fromthe node 300 a in place of the node 300 y. In either of these events,the active communications session extending between the D-modules of thenodes 300 a and 300 y may become inactive. In some embodiments,indications of such changes in which communication sessions are activeand/or inactive may be relayed to the one of the Managing modules 400that is in communication with the ingest server 2400 to enable thoseindications to be relayed onward to the ingest server 2400 alongsideindications of which communication sessions were originally configuredto be active, at least by default.

FIGS. 8A-D each illustrate a block diagram of a portion of an embodimentof the administration system 2000 of FIG. 1 in greater detail. Morespecifically, FIG. 8A depicts aspects of the operating environment of anexample embodiment of the ingest server 2400, FIG. 8B depicts aspects ofthe operating environment of an example embodiment of the databaseserver 2500, FIG. 8C depicts aspects of the operating environment of anexample embodiment of the visualization server 2600, and FIG. 8D depictsaspects of the operating environment of an example embodiment of one ofthe administration devices 200 or 2200.

Turning to FIG. 8A, in various embodiments, the ingest server 2400incorporates one or more of a processor component 2450, a clock 2455, amemory 2460 and an interface 2490 to couple the ingest server 2400 to atleast the network 999. The memory 2460 may store the account database2430 and a control routine 2440. The account database 2430 may be madeup of numerous ones of the account entries 2432, and each of the accountentries 2432 may include one or more system entries 2431. The controlroutine 2440 may incorporate a sequence of instructions operative on theprocessor component 2450 in its role as a main processor component ofthe ingest server 2400 to implement logic to perform various functionsduring execution of the control routine 2440 by the processor component2450.

As depicted, the control routine 2440 may incorporate a retrievalcomponent 2441 executable by the processor component 2450 to operate theinterface 2490 to receive information concerning the configuration andoperating aspects of the one or more storage cluster systems 1000 fromat least one node 300 of each. As depicted, it may be the managingmodule 400 of the at least one node 300 of each of the storage clustersystems 1000 that transmits the information concerning configuration andaspects of operation. As previously discussed, each of the storagecluster systems 1000 may vary greatly in complexity from relativelysimple embodiments that incorporate as little as a single node 300 and asingle storage device 800, to relatively complex embodiments thatincorporate multiple nodes 300 and numerous storage devices 800 coupledand configured to provide multiple forms of fault tolerance.

The retrieval component 2441 may operate the interface 2490 torecurringly contact the at least one node 300 of one or more of thestorage cluster systems 1000 via the network 999 to poll for suchinformation on what may be a regular interval. Alternatively oradditionally, the retrieval component 2441 may operate the interface2490 to await transmission of such information to the ingest server 2400by one or more of the storage cluster systems 1000. Again, one or moreof the storage cluster systems 1000 may transmit such information to theingest server 2400 at a recurring interval of time and/or in response tothe occurrence of one or more particular events as part of providing theingest server 2400 with a record thereof for subsequent diagnostics. Assuch information is recurringly received, the retrieval component 2544may employ indications of the current time and/or the current datemaintained by the clock 2455 to timestamp pieces of the receivedinformation to indicate the time and/or date at which each such piece isreceived.

The information so collected from each of the storage cluster systems1000 may include indications of various aspects of the hardware and/orsoftware components that make up each of the storage cluster systems1000, such as versions of those components and/or dates of manufactureof those components. Such information may include indications of themanner in which various aspects of each of the storage cluster systems1000 are configured, such as the manner in which various hardwarecomponents thereof are coupled and/or the manner in which client dataand/or other data are organized as stored within one or more of thestorage devices 800. Such information may include indications offeatures of each of the storage cluster systems 1000 that are enabledand/or disabled, as well as features of individual hardware and/orsoftware components, and as well as indications of the manner in whichone or more of those features are configured. Such information mayinclude indications of what applications software is used with each ofthe storage cluster systems 1000, including versions of thoseapplications, histories of changes in what applications are used, and/orhistories of the pattern and/or degree of usage of each of thoseapplications. Such information may include indications of the kind ofclient data stored within one or more of the storage devices 800 of eachof the storage cluster systems 1000, including types of data files,versions of the file types that are used, the sizes of various types ofdata files, and/or the pattern and/or frequency of accesses made tovarious types of data files. Such information may include indications ofoccurrences of various events within or otherwise involving each of thestorage cluster systems 1000, including types of events (e.g.,malfunctions, instances of exceeding storage capacity, resizing ofvolumes, additions and/or removals of storage devices 800, etc.), theoutcomes of various events, and/or the pattern and/or frequency ofoccurrence of various types of events. Such information may includeidentities and/or contact information for one or more administratorsassociated with an operator of one or more of the storage clustersystems 1000 (e.g., a network address of one of the administrationdevices 200 that is associated with one or more of thoseadministrators).

As also depicted, the control routine 2440 may incorporate a databasecomponent 2444 executable by the processor component 2450 to organizeand store such information as is received from the at least one node 300of each of the storage cluster systems 1000 in the account database2430. As previously discussed, the account database 2430 may be dividedinto multiple account entries 2432 with each of the account entries 2432storing all of such information received from one or more storagecluster systems 1000 that are operated by a single storage clustersystem operator. As also previously discussed, where a single storagecluster system operator operates multiple ones of the storage clustersystems 1000, the information received from each may be stored inseparate system entries 2431 defined within the account entry 2432associated with that storage cluster system operator. Within each ofsystem entries 2431, information may be organized chronologically.

Thus, upon the retrieval by the retrieval component 2441 of informationconcerning a particular storage cluster system 1000 may be time stampedby the retrieval component 2441, and then stored within the one of thesystem entries 2431 that corresponds to the particular storage clustersystem 1000 and that is maintained as part of the account entry 2432corresponding to the operator of that particular storage cluster system1000. Further, each new piece of information concerning that particularstorage cluster system 1000 may be chronologically organized within thatone of the system entries 2431 by the timestamp that may have been givento it by the retrieval component 2441 indicating when it was receivedand/or by an indication within that piece of information of when anevent involving that particular storage cluster system 1000 is indicatedwithin that piece of information to have occurred.

The information received by the ingest server 2400 may be a combinationof metadata (e.g., the example metadata 630 ab), configuration data onwhich metadata may be based and/or data indicating the details of anevent occurring within a storage cluster system 1000. In someembodiments, the retrieval component 2441 may timestamp the time and/ordate at which each such piece of data is received by the ingest server2400 and the database component 2444 store each such piece of datawithin an appropriate one of the system entries 2431 with little in theway of interpretation or modification.

In some embodiments, it may be the arrival of metadata and/orconfiguration data associated with a storage cluster system 1000 fromwhich no data has previously been received that may trigger thegeneration of a new system entry 2431, either within an existing accountentry 2432 if such a new storage cluster system 1000 is operated by aknown operator or within a new account entry 2432 generated to store thenew system entry 2431 if the operator of the new storage cluster system1000 is also new. Following the generation of a least a new system entry2431 for a new storage cluster system 1000, pieces of data that aresubsequently received that indicate the occurrence of events within thatnew storage cluster system 1000 may then be stored within that newsystem entry 2431.

In some embodiments, changes to the configuration of a storage clustersystem 1000 that entail the addition or removal of one or morecomponents may be conveyed to the ingest server 2400 by the receipt ofnew metadata and/or configuration data that is intended to replace orotherwise override earlier received metadata and/or configuration data.In some embodiments, the database component 2444 may compare thecontents of more recently received metadata and/or configuration data toearlier received pieces of such data to identify configuration changesthat may have taken place in a storage cluster system 1000.Alternatively or additionally, such changes made in a storage clustersystem 1000 may be indicated as events in much the same manner asfailure events and still other types of events in pieces of datareceived by the ingest server 2400 that convey the details of events.

Turning to FIG. 8B, in various embodiments, the database server 2500incorporates one or more of a processor component 2550, a memory 2560and an interface 2590 to couple the database server 2500 to at least thenetwork 999. The memory 2560 may store the system database 2530,organizational data 2535 and a control routine 2540. The system database2530 may be made up of numerous ones of the system entries 2531, andeach of the system entries 2531 may include multiple object entries2533, event entries 2536 and/or relationship entries 2539. The controlroutine 2540 may incorporate a sequence of instructions operative on theprocessor component 2550 in its role as a main processor component ofthe database server 2500 to implement logic to perform various functionsduring execution of the control routine 2540 by the processor component2550.

As depicted, the control routine 2540 may incorporate a retrievalcomponent 2544 executable by the processor component 2550 to operate theinterface 2590 to recurringly access the account entries 2432 of theaccount database 2430 maintained by the ingest server 2400. Theretrieval component 2544 may operate the interface 2590 to recurringlycontact the ingest server 2400 via the network 999 to poll for thecontents of each of the account entries 2432 on what may be a regularinterval. Alternatively or additionally, the retrieval component 2544may operate the interface 2590 to await transmission of the contents ofeach of the account entries 2432 by the ingest server 2400, which maytransmit such contents to the database server 2500 at a recurringinterval of time and/or in response to the occurrence of one or moreparticular changes to the contents of one of the account entries 2432.

As also depicted, the control routine 2540 may incorporate a databasecomponent 2545 executable by the processor component 2550 to generateand maintain the system entries 2531 of the system database 2530. Aspreviously discussed, within each system entry 2531, each of variouscomponents of a storage cluster system 1000 may be represented as anobject in which various details of its operation and/or its interactionswith other components within a storage cluster system 1000 may berepresented as properties of that object. As part of instantiating suchan object, the database component 2545 may allocate a portion of thememory 2560 in which to create an object entry 2533 made up of one ormore data structures in which is stored indications of various detailsof the component for which the object is instantiated.

In instantiating objects for each of various components of a storagecluster system, the database component 2545 may refer to theorganizational data 2535, which may specify what components of a storagecluster system 1000 are to be represented with an object and/or variousaspects of the manner in which an object is to be instantiated and/orrepresented within the memory 2560. By way of example, whether hardwareand/or software components making up a single node 300 are to each beindividually represented with an object in addition to or lieu of thatentire single node 300 being represented with an object may be specifiedwithin the organizational data 2535. Also, the organizational data 2535may specify the data structure(s) that are to be defined within eachobject entry 2533 as part of instantiating an object within the memory2560. As part of specifying one or more of such data structures, theorganizational data 2535 may specify the amount of space that is to beallocated within the memory 2560 for each object entry 2533.

In some embodiments, relationships between components that are sorepresented as objects may be treated as part of the properties of thoseobjects. Thus, in such embodiments, each of the object entries 2533 mayinclude indications of relationships that the associated component haswith one or more other components. Such relationships may include aconnection to another component via an interconnect (e.g., one of theexample interconnects 399, 599 a, 699 ab, 899 ab, etc.). Alternativelyor additionally, such relationships may include a component beingincorporated into another component such that one component is actuallypart of another component. In other embodiments, and as depicted, eachrelationship between two or more components may be represented with arelationship entry 2539 instantiated in a manner analogous to theinstantiation of the object entries 2533, which may include indicationsof the type of relationship among two or more components and/or whichcomponent is at a higher or lower level relative to the other(s) wherethe relationship is of a hierarchical nature.

FIG. 9 depicts a portion of an example embodiment of representingcomponents of the example storage cluster system 1000 of FIGS. 3 through7 as objects. It should be noted that FIG. 9 is in no way an exhaustivedepiction of representing components as objects as none of the objectswithin the cluster 1300 z, the HA group 1600 yz, the node 300 b, etc.are depicted as being represented as objects. Instead, FIG. 9 depictsthe manner in which components of the set of storage devices 800 ab andthe node 300 a may be represented as objects, as well as the storagecluster system 1000, the cluster 1300 a, the HA group 1600 ab, the node300 a and the storage device 800 ab may be represented as objects. Asindicated in FIG. 9, each such object may be defined by an object entry2533 generated within space allocated for it within the memory 2560.

Also depicted in FIG. 9 are indications of relationships betweendifferent ones of the objects. For example, the relationship of the node300 a as part of the HA group 1600 ab, the relationship of the HA group1600 ab as part of the cluster 1300 a, and the relationship of thecluster 1300 a as part of the storage cluster system 1000 are depicted.Also depicted is the coupling of the set of storage devices 800 ab tothe data module 600 of the node 300 a via the storage interconnect 899ab. Again, and as previously discussed, relationships between objectsmay be treated as part of the set of properties of those objects and maytherefore be indicated among the indications of properties included inthe object entries 2533 associated with those objects. Alternatively,and as depicted in FIG. 9, indications of relationships between objectsmay be separately stored in a relationship entry 2539 for each suchrelationship. Regardless of exactly what components of a storage clustersystem 1000 are specified in the organizational data 2535 to berepresented as objects, and regardless of the exact manner in whichindications of relationships between objects are stored and/or whatrelationships are the ones for which indications are stored, thedatabase component 2545 effectively models each storage cluster system1000 as a combination of objects linked by relationships therebetween.

Returning to FIG. 8B, as database server 2500 receives information fromthe ingest server 2400 indicating that changes have occurred to one ormore components, the database component 2545 may access the one or moreobject entries 2533 that correspond to those one or more components, andmay alter one or more indications of properties in those object entries2533 to provide indications of those changes. In some embodiments, theindications of properties within each of the object entries 2533 may beat least partly organized chronologically and/or may be timestamped toenable a chronological ordering of changes made to one or more of theproperties as a mechanism to store a record of those changes made overtime. Alternatively or additionally, where the changes indicated ininformation received from the ingest server 2400 include the addition ofcomponents to a storage cluster system 1000, the database component 2545may generate more object entries 2533 as part of instantiating newobjects to represent those additional components. In so doing, anindication of when a component was so added may be included among theproperties stored in the corresponding object entry 2533. Alsoalternatively or additionally, where the changes indicated ininformation received from the ingest server 2400 include the removal ofa component from a storage cluster system 1000, the database component2545 may add an indication of that component having been removed as wellas an indication of when the removal occurred to the properties includedin the corresponding object entry 2533. Subsequently, the indication ofthe component having been removed may be treated by the databasecomponent 2545 as an indication that the object representing thatremoved component has itself been removed.

As an alternative to chronologically storing indications of eventsassociated with particular objects as part of the properties of theobject entries 2533 associated with those objects, the databasecomponent 2545 may store indications of events in individual ones of theevent entries 2536. The event entries 2536 may be organized in a mannerenabling the retrieval and stitching together of all events occurring ata specific time and/or occurring within a specific range of times. Thismay at least partially enable the state of all objects of a storagecluster system at a specified time and/or within a specific range oftimes to be determined. Alternatively and/or additionally, the evententries 2536 may be organized in a manner enabling the retrieval andstitching together of all events associated with a specified object.This may at least partially enable changes in the state of the specifiedobject over a range of time and/or between two specified times to bedetermined In embodiments in which indications of events that change thestate of one or more objects are stored separately from the objectentries 2533, such as where indications of such events are stored in theevent entries 2536, the properties of each of the object entries 2533may then store indications of only the current state of the objects towhich they correspond.

As further depicted, the database component 2545 may include a searchcomponent 2546 to respond to a request for information concerning aspecific object and/or specified neighbors of a specific object bysearching the system database 2530 for those object(s) and retrievingindications of their properties. Where a request concerns only aspecific object, the search component 2546 may retrieve and provideindications of the properties of that object, alone, and thoseproperties may include indications of relationships of that object toother objects. Where those indications of relationships are stored asproperties of the object entry 2533 for that object, such retrieval mayentail merely retrieving those indications from that object entry 2533.However, where those indications of relationships are separately storedin one or more relationship entries 2539, then the search component 2545may search the relationship entries 2539 for the indications of thoserelationships in addition to searching for the one object entry 2533that corresponds to the specified object.

However, where the request additionally concerns neighboring objectshaving relationships with the specific object, and where the indicationsof relationships to other objects are included as properties of thatobject that are stored in the object entry 2533 for that object, thenthe search component 2546 may retrieve those indications ofrelationships from that object entry 2533 and may use those indicationsto identify the other objects that are specified as also included in therequest. The search component 2546 may then retrieve the object entries2533 for those neighboring objects to retrieve the indications of theproperties thereof to include in the response to the request.Alternatively, where the indications of relationships between objectsare stored as separate relationship entries 2539, the search component2546 may search among the relationship entries 2539 for ones of therelationship entries 2539 corresponding to relationships that areassociated with the specified object, and may then use the indicationsof the properties of each of those relationships to identify the otherobjects that are specified as also included in the request.

Turning to FIG. 8C, in various embodiments, the visualization server2600 incorporates one or more of a processor component 2650, a memory2660 and an interface 2690 to couple the visualization server 2600 to atleast the network 999. The memory 2660 may store rendering data 2633,practices data 2634 and a control routine 2640. The control routine 2640may incorporate a sequence of instructions operative on the processorcomponent 2650 in its role as a main processor component of thevisualization server 2600 to implement logic to perform variousfunctions during execution of the control routine 2640 by the processorcomponent 2650.

As depicted, the control routine 2640 may incorporate a communicationscomponent 2649 to interact with one or more administration devices 200and/or 2200 through the network 999 to enable the one or moreadministration devices 200 and/or 2200 to present visualizations of atleast a portion of a storage cluster system 1000. More specifically, thecommunications component 2649 may receive requests from the one or moreadministration devices 200 and/or 2200 for information concerning atleast a portion of a portion of a storage cluster system 1000. Thecommunications component 2649 may respond by transmitting either therequested information following its retrieval from the system database2530 to the one or more administration devices 200 and/or 2200, or mayrespond by transmitting a visualization based on that requestedinformation thereto. In some embodiments, the communications componentmay implement a web server to exchange commands and/or data in ascripting language such as hypertext markup language (HTML) and/or otherlanguage(s) with the one or more administration devices 200 and/or 2200as a mechanism to receive requests therefrom and to transmit requestedinformation and/or visualizations thereto.

As also depicted, in embodiments in which the visualization server 2600may itself generate visualizations of at least a portion of a storagecluster system 1000 that corresponds to and visually depicts informationrequested by an administration device 200 or 2200, the control routine2640 may incorporate a rendering component 2643 to do so. Morespecifically, the rendering component 2643 may render an image withinthe visualization server 2600 (e.g., within a portion of the memory2660) that includes a visualization that depicts the requestedinformation. The communications component 2649 may transmit a bitmap,vector display list or other form of representation of thatvisualization to the that one of the administration devices 200 or 2200for it to then present on a display. In so doing, the renderingcomponent 2643 may refer to the rendering data 2633, which may specifyvarious parameters for how components and/or relationships betweencomponents are to be represented in a visualization, and/or may includepre-rendered bitmap or vector-based images of components and/orrelationships between components to be used in generating avisualization. Alternatively or additionally, in so doing, the renderingcomponent 2643 may refer to the practices data 2634, which may specifyvarious criteria for determining whether a state of a component and/or arelationship between components represents a risk to be highlighted in avisualization, and/or which may specify various aspects of suggestedcourses of action to be highlighted in a visualization in response tosuch risks.

As further depicted, the control routine 2640 may incorporate atranslation component 2645 to translate requests received from one ormore administration devices 200 and/or 2200 for information concerningat least a portion of a storage cluster system 1000 into one or moresets of query instructions that may be used in searching differentportions of the system database 2530 for that information. As previouslydiscussed in reference to FIG. 8B, each system entry 2531 of the systemdatabase 2530 may be made up of multiple types of other entries thateach correspond to different portions and/or different aspects of astorage cluster system 1000 associated with that system entry 2531. Morespecifically, a system entry 2531 may include multiple object entries2533 that each correspond to a different component of a storage clustersystem 1000, multiple event entries 2536 that each correspond to adifferent event that may be associated with one or more of thecomponents of that storage cluster system 1000, and/or multiplerelationship entries 2539 that each correspond to an interconnectbetween components of that storage cluster system 1000 or an instance ofone component of that storage cluster system 1000 being incorporatedinto another. Thus, a requested for information received from anadministration device 200 or 2200 may be translated by the translationcomponent 2645 into query instructions to gain access to the systementry 2531 for a particular cluster, and then into one or more separatesets of query instructions to obtain information from one or more ofeach of the object entries 2533, one or more of the event entries 2536and/or one or more of the relationship entries 2539. The translationcomponent 2645 may provide those different sets of query instructions tothe search component 2546 within the database server 2500 via thenetwork 999 to be executed by the search component 2546 to obtain therequested information. Upon receiving such requested information fromthe database server 2500, the translation component 2645 may assemblethe requested information from the different queries into a single setof information that the translation component 2645 may then relay to thecommunications component 2649 to transmit onward to the administrationdevice 200 or 2200 that requested that information. Alternatively oradditionally, the translation component 2645 may relay the informationto the rendering component 2643 to use in generating a visualization.

Turning to FIG. 8D, in various embodiments, each of the administrationdevices 200 or 2200 incorporates one or more of a processor component250, a memory 260, an input device 220, a display 280, and an interface290 by which to be coupled to at least the network 999. The memory 260may store a control routine 240. The control routine 240 may incorporatea sequence of instructions operative on the processor component 250 inits role as a main processor component of one of the administrationdevices 200 or 2200 to implement logic to perform various functionsduring execution of the control routine 240 by the processor component250.

As depicted, the control routine 240 may incorporate a communicationscomponent 249 to operate the interface 290 to establish any of a varietyof types of communications with the visualization server 2600. In someembodiments, the communications component 249 may transmit one or moresecurity credentials to the visualization server 2600 to establish thatthe administration device 200 or 2200 is an accepted device for use inproviding visualizations of a storage cluster system 1000 and/or that anadministrator operating it is authorized to have access to suchvisualizations.

As also depicted, the control routine 240 may incorporate a userinterface (UI) component 248 to operate the input device 220 and/or thedisplay 280 to provide an administrator operating the administrativedevice 200 or 2200 with a UI by which they may interact to providerequests for visualizations of a storage cluster system 1000. In someembodiments, the UI component 248 may operate the display 280 to providea text editing environment into which an administrator may manuallyenter a command requesting information concerning a particular storagecluster system 1000 that the administrator is tasked with overseeing aspart of obtaining at least an initial visualization thereof. The text soentered by an administrator may then be transmitted by thecommunications component 249 to the visualization server 2600.

As further depicted, the control routine 240 may incorporate aninterpretation component 242 to cooperate with the UI component 248 tointerpret operation of the input device 220 by an administrator as partof interacting with menus and/or images of objects that the UI component248 may present on the display 280. By way of example, instead of the UIcomponent 248 operating the display 280 to present a command prompt ortext editor to enable text entry of a command requesting information byan administrator, the UI component 248 may operate the display 280 topresent a menu of options that the administrator may interact withthrough operation of the input device 220. More specifically, the UIcomponent 248 may present on the display 280 a menu populated withselectable menu items that each correspond to a different storagecluster system 1000 from which the administrator may select a particularstorage cluster system 1000 to be presented with a visualization of.Further, the UI component 248 may also present a graphical pointer onthe display 280 (e.g., a mouse pointer) that may be manipulated throughoperation of the input device 220 to effect such a selection where theinput device 220 includes a form of pointer device (e.g., a mouse, atouchpad, a trackball, a joystick, etc.).

The interpretation component 242 may respond to a selection of a menuitem corresponding to a particular storage cluster system 1000 by itselfgenerating the text of a command to request information concerning theparticular storage cluster system 1000 that has just been selected aspart of preparing to present the requested visualization on the display280. This automatically generated text may then be transmitted to thevisualization server 2600 by the communications component 249 just asmanually entered text of the same call would be. As previouslydiscussed, in response to a request for information concerning, whetheran initial visualization of an entire storage cluster system 1000 or asubsequent visualization of a portion thereof, the visualization server2600 may respond by providing either data representing a visualizationas already generated by the visualization server 2600 (e.g., a bitmap ora vector display list representing a visualization). The communicationscomponent 249 may relay such a representation of a visualization to theUI component 248 to enable the UI component 248 to present thevisualization on the display.

As still further depicted, in embodiments in which an administrationdevice 200 or 2200 may itself generate visualizations of at least aportion of a storage cluster system 1000 based on information requestedthereby, the control routine 240 may incorporate a rendering component243 to do so. The communications component 249 may relay such receivedinformation to the rendering component 243, and the rendering component243 may employ the information to generate a visualization of at least aportion of that storage cluster system 1000 (e.g., render thevisualization into a portion of the memory 260) and then present thatvisualization on the display 280.

With a visualization presented of at least a portion of a storagecluster system 1000 presented on the display 280, the UI component 248may monitor further operation of the input device 220 for indications ofan administrator operating it to select menu items and/or to selectportions of the visualization as part of providing a request for anothervisualization of a different scope than the one currently presented.Again, the interpretation component 242 may cooperate with the UIcomponent 248 to interpret such actions by the administrator and/or togenerate new commands to provide to the visualization server 2600. Insome embodiments, the UI component 248, the interpretation component 242and/or the rendering component 243 may be components of a web browser tointeract with the visualization server 2600 as a web server.Alternatively or additionally, it may be that the interpretationcomponent 242 is a so-called “add-on” or “plug-in” that augments the UIcomponent 248 with ability to so interpret operation of the input device220 by an administrator and to automatically generate commands to betransmitted to the visualization server 2600. Also alternatively oradditionally, it may be that the rendering component 243 is such anadd-on or plug-in that augments the UI component 248 with the ability torender such visualizations.

However, in embodiments in which the rendering component 2643 of thevisualization server 2600 generates visualizations of at least a portionof a storage cluster system 1000 based on information requested by anadministration device 200 or 2200, the control routine 2640 mayincorporate an interpretation component 2642 to interpret actions by anadministrator operating that administration device 200 or 2200, and/orto generate new commands within the visualization server 2600.Regardless of whether commands are generated remotely by theinterpretation component 242 within the administration device 200 or2200, or are generated locally by the interpretation component 2642within the visualization server 2600, again, those commands may requestinformation by making reference to the components of a storage clustersystem 1000 as objects in keeping with the object-based model by whichinformation about a storage cluster system 1000 is stored within eachsystem entry 2531 of the system database 2530. Where such use is made ofthe rendering component 2643 and/or of the interpretation component2642, it may be that the communications component 2649 of thevisualization server 2600 interacts with the administration device 200or 2200 in the manner of a client-server relationship in which theadministration device 200 or 2200 is treated as a terminal of thevisualization server 2600. Alternatively or additionally, thecommunications component 2649 of the visualization server 2600 mayinteract with the administration device 200 or 2200 as a web serverproviding each visualization of a storage cluster system 1000 as awebpage and/or as a portion of a webpage (e.g., as a bitmap image of avisualization, a vector-based representation of an image of avisualization, etc.). In either of such types of interaction, theinterpretation component 2642 may remotely receive indications from theUI component 248 of operation of the administration device 200 or 2200by an administrator to select a menu item and/or to select a portion ofa visualization that may include indications of where a portion of amenu and/or a visualization has been selected or otherwise interactedwith.

It should be noted that although FIG. 8A-D depict each of the servers2400, 2500 and 2600 as separate and distinct computing devices withseparate and distinct processor components and/or memories, otherembodiments are possible in which two or more of the servers 2400, 2500and 2600 may be combined into a single server. By way of example, aprocessor component of a single server may execute the instructions ofboth the control routines 2440 and 2540, or of both the control routines2540 and 2640. Also by way of example, two or more of the servers 2400,2500 and 2600 may be implemented as virtual machines generated within asingle server by at least one processor component of that single server.

FIGS. 10A-D each illustrate an example embodiment of an administrationdevice 200 or 2200 generating commands to request information from thesystem database 2530 through the visualization server 2600 of theexample administration system 2000 of FIGS. 8A-8D as part of presentingvisualizations of at least portions of an example storage cluster system1000 in greater detail. More specifically, FIG. 10A depicts aspects ofgenerating a request to retrieve information for presenting an initialvisualization of a whole storage cluster system 1000, FIG. 10B and FIG.10C each depicts aspects of different approaches to generating a requestto retrieve information for presenting a new visualization of a portionof that storage cluster system 1000, and FIG. 10D depicts aspects ofrendering such a new visualization.

Turning to FIG. 10A, the UI component 248 may have generated a menu 282of selectable menu items 281 that is presented on the display 280 of anadministration device 200 or 2200. The depicted menu items 281 eachcorrespond to a different one of multiple storage cluster systems 1000for which one or more visualizations may be requested. The depictedidentifiers and/or other information specifying the storage clustersystems 1000 that are available to be selected, may have been providedto the administration device 200 or 2200 by the visualization server2600 for use by the UI component 248 in generating the menu items 281 ofthe menu 282 from the system entries 2531 of the system database 2530.

As depicted, an administrator tasked with overseeing operation of astorage cluster system 1000 operated by “JKL” corporation for its NorthAmerican (“NA”) branch has operated the administration device 200 or2200 (e.g., by manually operating the input device 220) to use anon-screen pointer 288 to select the menu item 281 that corresponds tothat storage cluster system 1000, thereby requesting an initialvisualization thereof. The interpretation component 242 detects theselection of that menu item 281, and automatically generates a commandrequesting information concerning that storage cluster system 1000. Insome embodiments, the interpretation component 242 may generate thatcommand to specify the scope of the requested information to includeinformation covering the current state of all of the components of thatstorage cluster system 1000. The specifying of such a scope of therequested information may be a default scope employed automatically bythe interpretation component 242 in some embodiments.

Following its generation, the command requesting such information may betransmitted to the communications component 2649 of the visualizationserver 2600 via the network 999, where the command may be relayed to thetranslation component 2645 thereof. The translation component 2645 maytranslate the request for information, as well as the scope it specifiesfor that information, into database query instructions executable by thesearch component 2546 to first locate the system entry 2531 for theselected storage cluster system 1000 (specifically, the storage clustersystem “JKL NA”), and then to retrieve indications of the current statusfrom all of the object entries 2533 and/or the relationship entries 2539within that system entry 2531.

Turning to FIG. 10B, the UI component 248 may have generated anothermenu 282 of selectable menu items 281 that is presented on the display280 of an administration device 200 or 2200, along with a timelinescroll bar 286. Also, in embodiments in which visualizations aregenerated by the administration devices 200 and/or 2200, the renderingcomponent 243 may have generated a visualization 780 of the particularstorage cluster system 1000 selected in FIG. 10A. As depicted, thevisualization 780 is made up of selectable object items 783 that eachcorrespond to an object of the particular storage cluster system 1000and selectable relationship items 789 that each correspond to arelationship between two objects that are each visually presented withone of the selectable object items 783. As depicted, the selectableobject item 783 at the root in this example visualization corresponds tothe object that represents the entirety of the particular storagecluster system 1000, while each of the selectable object items 783furthest from the root correspond to an object that represents one ofthe individual LUNs 862 of the particular storage cluster system 1000.

As also depicted, one of the selectable object items 783 has beenselected through use of the on-screen pointer 288 as a mechanism toselect a particular object to be depicted in a requested newvisualization of only a portion of the particular storage cluster system1000. Within the menu 282, the selectable menu items provide a mechanismto specify how large a neighborhood of other objects surrounding theparticular object are to also be included in the new visualization. Insome embodiments, objects that are closer to the root than theparticular object may be deemed to be at level(s) of the object-basedmodel of the particular storage cluster system 1000 that are outside theparticular object, while objects that are further away from the rootthan the particular object may be deemed to be at level(s) of theobject-based model of the particular storage cluster system 1000 thatare inside the particular object.

The concept of “outside” and “inside” relative to an object may be basedon the occurrence of relationships between objects in which one objectrepresents a component that is incorporated into another componentrepresented by the other object such that the one component is deemed tobe “inside” the other component. Such a relationship may be depictedwith a selectable relationship item 789 (e.g., the depicted linesegments) extending between the selectable object items 783 for each ofthe two components, with the one component that is incorporated into theother being positioned further away from the root by the one level ofrelationship represented by that selectable relationship item 789. Asdepicted, the two menu items 281 have been operated in a manner tospecify that neighboring objects that are one level outside and onelevel inside the particular object that has been selected are to beincluded in the new visualization.

As further depicted, within the timeline scroll bar 286, a pair of timepoints T1 and T2 have been selected as part of requesting that the newvisualization depict how the states of objects and relationshipstherebetween have changed between the time points T1 and T2. Thus, whilethe visualization 780 already presented on the display 280 is based oninformation concerning the state of the objects and relationshipsbetween objects of the particular storage cluster system 1000 at asingle particular time (e.g., the current time), the new visualizationbeing requested by an administrator through such use of the timelinescroll bar 286 is to be based on information concerning the states ofobjects and relationships between objects of the particular storagecluster system 1000 at two particular time points T1 and T2.

The interpretation component 242 detects the selection of the particularsubset of objects and relationships between objects through theselection of the particular selectable object 783 and through theselection of only neighboring objects that are within one outside andonly one inside level via the selectable menu items 281. Theinterpretation component 242 may also detect the selection of the twotime points T1 and T2 via the timeline scroll bar 286, and automaticallygenerates a command requesting information with a scope that includesthe object that corresponds to the selected one of the selectable objectitem 783, any objects within the selected number of levels outside ofthe selected object, and any objects within the number of levels insideof the selected object. The scope may also specify that the informationis to span a period of time defined by the time points T1 and T2.

Following its generation, the command requesting such information may betransmitted to the communications component 2649 of the visualizationserver 2000 via the network 999, where the command may be relayed to thetranslation component 2645. The translation component 2645 may translatethe request for information, as well as the scope it specifies for thatinformation, into database query instructions executable by the searchcomponent 2546 to first locate the system entry 2531 for the selectedstorage cluster system 1000 (specifically, the storage cluster system“JKL NA”), and then to retrieve indications of the current status fromall of the object entries 2533 and/or the relationship entries 2539within that system entry 2531. The translation component 2645 translatesthese indications of a request for information concerning this morelimited set of objects into database query instructions executable bythe search component 2546 to first locate the system entry 2531 for theparticular storage cluster system 1000 (specifically, the “JKL NA”storage cluster system), and then to retrieve information from a subsetof the object entries 2533 and/or the relationship entries 2539 withinthat system entry 2531 that correspond to the more limited set ofobjects and the relationships thereamong. The translation component 2645also translates the indications of a request for information concerningthe states of this more limited set of objects at the particular timesT1 and T2 into query instructions executable by the search component2546 to also retrieve information from the event entries 2536 thatprovide indications of events that may have defined the state of thismore limited set of objects and relationships at each of the times T1and T2.

As recognizable to those skilled in the art, any of a variety of datastructures may be used to organize the data making up each system entry2531 of the system database 2530, and as a result, the manner in whichthe data within a single system entry 2531 may need to be searched toretrieve the states of some or all of the objects associated with astorage cluster system 1000 at a time earlier than the current time mayvary greatly. By way of example, if each of the object entries 2533includes indications of only the current state of a correspondingobject, then it may be necessary to search through the ones of the evententries 2536 detailing events that affect a particular object startingwith the most recent of those events and proceeding backwards into thepast to that earlier time. Also, as each such event is found whileproceeding backwards into the past, the changes in state made by eachsuch event to that particular object may need to be analyzed to derivethe state of that object at that earlier time.

FIG. 10C depicts an alternate approach to what was depicted in FIG. 10Bfor the selection of a subset of selectable objects 783 and/orselectable relationship items 789 to be included in a new visualization.Similar to what was depicted in FIG. 10B, the UI component 248 may havegenerated the same timeline scroll bar 286, and the rendering component243 may have generated a visualization 780 of the particular storagecluster system 1000 selected in FIG. 10A. Also similar to what wasdepicted in FIG. 10B, the selectable object item 783 at the root in thisexample visualization corresponds to the object that represents theentirety of the particular storage cluster system 1000, while each ofthe selectable object items 783 furthest from the root correspond to anobject that represents one of the individual LUNs 862 of the particularstorage cluster system 1000.

However, unlike what was depicted in FIG. 10B, the UI component 248 maynot have generated the same menu 282 with the same selectable menu items281 for use in specifying how large a neighborhood of other objectsaround the particular selected object are to also be included in the newvisualization. Instead, the interpretation component 242 may support theuse of the on-screen pointer 288 by an administrator operating theadministration device 200 or 2200 to draw a box 289 or other form ofencircling graphic around the selected one of the selectable objectitems 783 (which corresponds to the particular selected object) todefine how large a neighborhood of other objects to include in the newvisualization by encircling their corresponding ones of the selectableobject items 783. By analyzing what selectable object items 783 are soencircled, the interpretation component 242 may derive the quantities oflevels inside and/or outside the particular selected object that are tobe included in the new visualization.

Similar to what was depicted in FIG. 10B, the interpretation component242 may also detect the selection of the two time points T1 and T2 viathe timeline scroll bar 286, and automatically generates a commandrequesting information with a scope that includes the object thatcorresponds to the selected one of the selectable object item 783, anyobjects within the selected number of levels outside of the selectedobject, and any objects within the number of levels inside of theselected object. The scope may also specify that the information is tospan a period of time defined by the time points T1 and T2.

Turning to FIG. 10D, a new menu 282 of selectable menu items 281 and anew timeline scroll bar 286 are presented on the display 280 of anadministration device 200 or 2200, along with the new visualization 780of a portion of the storage cluster system 1000 that was requested viathe actions described in either of FIG. 10B or 10C. The newvisualization 780 is also made up of selectable object items 783 thateach correspond to an object of the particular storage cluster system1000 and selectable relationship items 789 that each correspond to arelationship between two objects that are each visually presented withone of the selectable object items 783.

FIG. 10D additionally depicts the manner in which color and/or symbolsmay be employed by the rendering component 243 in generating the newvisualization 780, the menu 282 and/or the timeline scroll bar 287 tocommunicate more detail in each and/or to present correlationsthereamong. By way of example in the new visualization 780, differentcolors (depicted with differing cross-hatching patterns) may be used toindicate different types of events that have occurred to differentobjects and/or relationships between objects during the period of timedefined by the time points T1 and T2 to various. More specifically,through use of differing colors and a circular symbol denoting nodes inrendering different ones of the selectable object items 783, one node isdepicted as having been removed and another node is depicted as havingbeen added during that period of time. Also so depicted through use ofdifferent colors in rendering different ones of the selectablerelationship items 789 is the corresponding loss of the relationship ofthe removed node as a component of a HA pair and as a controller of anaggregate of that HA pair, as well as the formation of the relationshipof the added node as becoming a component of that HA pair and asbecoming a controller of that aggregate.

As further depicted, explanatory text within the menu 282 and/or thetimeline scroll bar 286 may serve as a legend that sets forth themeanings of each of the colors and/or each of the symbols used withinthe new visualization 780. FIG. 10D further depicts the manner in whichdifferent types of events may be presented at the times at which theyoccurred and/or with a separation between types of events that enableeach of the different types of events to be more easily focused upon.More specifically, through use of differing colors and/or throughdiffering vertical placement of event items 287 at different times alongthe timeline scroll bar 286

FIGS. 11A-D each illustrate an example embodiment of rendering andpresenting of modified forms of the new visualization of 780 of FIG. 10Din response to various ways in which the administration device 200 or2200 may be operated by an administrator to request the presenting ofmore or different information on the display 280. More specifically,FIG. 11A the depicts the selection of a single selectable object item783 to request further information concerning its corresponding object,FIG. 11B and FIG. 11C each depicts the selection of an item within themenu 282 and/or the timeline scroll bar 286 to control what informationis presented, FIG. 11D depicts selection of an alternate time point toalter temporal aspects of what is presented. It should be noted that,like FIG. 10D, the specific depiction of the interpretation component242, the UI component 248 and the rendering component 243 of anadministration device 200 and/or 2200 has been omitted for sake ofvisual clarity in each of FIGS. 11A-D, along with the specific depictionof components of the visualization server 2600 and the database server2500. FIGS. 10A-C should be referred to for depictions of examples ofthese components where needed or desired to understand what is describedwith regard to FIGS. 11A-D.

Turning to FIG. 11A, as depicted an administrator has operated theadministration device 200 or 2200 (e.g., by manually operating the inputdevice 220) to use the on-screen pointer 288 to select a selectableobject item 783 that corresponds to a node that was added to HA pair torequest the presentation of more information concerning that node. Theinterpretation component 242 detects the selection of that selectableobject item 783, and automatically generates a command requesting theadditional information concerning that corresponding node fortransmission to the visualization server 2600. The information requestedmay be retrieved from the indications of properties from the objectentry 2533 that corresponds to that node. Upon receipt of the requestedinformation from the visualization server 2600, the rendering component243 may augment its rendering of the new visualization 780 to include atextual presentation of the requested information, as depicted.

FIGS. 11B and 11C each depict alterations made by the renderingcomponent 243 to the new visualization 780 to reduce the amount ofinformation presented therein in response to selections of items withinthe menu 282 and/or the timeline scroll bar 286. Specifically, in FIG.11B, the rendering component 243 may modify the new visualization 780 toreduce the visual emphasis of all other items within the newvisualization 780 that don't represent nodes in response to theselection of the menu item 281 within the menu 282 that corresponds tonodes. Such reduction of visual emphasis may entail rendering the otherselectable object items 783 and/or the selectable relationship items 789to have a partially transparent quality to let through a backgroundcolor, or to re-render the new visualization 780 to remove the otherselectable object items 783 and/or the selectable relationship items 789entirely. In FIG. 11C, the rendering component 243 may similarly modifythe visualization 780 to reduce the visual emphasis of all other itemswithin the new visualization 780 that are not selectable object items783 and/or selectable relationship items 783 that don't representcomponents and/or relationships between components, respectively, thatwere removed during the period of time defined by the time points T1 andT2.

Turning to FIG. 11D, as depicted an administrator has operated theadministration device 200 or 2200 to use the on-screen pointer 288 toselect the time point T1 and to also select another time point that iswithin the period of time defined by the time points T1 and T2, therebyeffectively defining a subset of that period of time as a new period oftime for which the administrator is requesting a modified version of thenew visualization 780. The interpretation component 242 may detects theselection of the time point T1 and the new time point within thetimeline scroll bar 286, and automatically generates a new commandrequesting information with a scope that includes the same objects as inthe request that enabled the generation of the new visualization 780,but that also specifies that the information is to span this new periodof time. Upon receipt of the requested information from thevisualization server 2600, the rendering component 243 render such amodified version of the visualization 780 in much the same manner inwhich it earlier rendered the new visualization 780, itself. As aresult, and as depicted, the modified version of the new visualizationdepicts the removal of the node and associated relationships thatoccurred between the time point T1 and the new time point, but does notdepict the addition of the other node and associated relationships thatoccurred between the new time point and the time point T2.

In various embodiments, each of the processor components 450, 550, 650,2450, 2550 and 2650 may include any of a wide variety of commerciallyavailable processors. Also, one or more of these processor componentsmay include multiple processors, a multi-threaded processor, amulti-core processor (whether the multiple cores coexist on the same orseparate dies), and/or a multi processor architecture of some othervariety by which multiple physically separate processors are in some waylinked.

In various embodiments, one or more of the control routines 440, 540,640, 2440, 2540 and 2640 may include one or more of an operating system,device drivers and/or application-level routines (e.g., so-called“software suites” provided on disc media, “applets” obtained from aremote server, etc.). As recognizable to those skilled in the art, eachof the control routines 440, 540 and 640, including the components ofwhich each may be composed, are selected to be operative on whatevertype of processor or processors may be selected to implement applicableones of the processor components 450, 550 or 650, or to be operative onwhatever type of processor or processors may be selected to implement ashared processor component. In particular, where an operating system isincluded, the operating system may be any of a variety of availableoperating systems appropriate for corresponding ones of the processorcomponents 450, 550 or 650, or appropriate for a shared processorcomponent. Also, where one or more device drivers are included, thosedevice drivers may provide support for any of a variety of othercomponents, whether hardware or software components, of correspondingones of the modules 400, 500 or 600.

In various embodiments, one or more the memories 460, 560, 660, 2460,2560 and 2660 may be based on any of a wide variety of informationstorage technologies, possibly including volatile technologies requiringthe uninterrupted provision of electric power, and possibly includingtechnologies entailing the use of machine-readable storage media thatmay or may not be removable. Thus, each of these memories may includeany of a wide variety of types (or combination of types) of storagedevice, including without limitation, read-only memory (ROM),random-access memory (RAM), dynamic RAM (DRAM), Double-Data-Rate DRAM(DDR-DRAM), synchronous DRAM (SDRAM), static RAM (SRAM), programmableROM (PROM), erasable programmable ROM (EPROM), electrically erasableprogrammable ROM (EEPROM), flash memory, polymer memory (e.g.,ferroelectric polymer memory), ovonic memory, phase change orferroelectric memory, silicon-oxide-nitride-oxide-silicon (SONOS)memory, magnetic or optical cards, one or more individual ferromagneticdisk drives, or a plurality of storage devices organized into one ormore arrays (e.g., multiple ferromagnetic disk drives organized into aRAID array). It should be noted that although each of these memories isdepicted as a single block, one or more of these may include multiplestorage devices that may be based on differing storage technologies.Thus, for example, one or more of each of these depicted memories mayrepresent a combination of an optical drive or flash memory card readerby which programs and/or data may be stored and conveyed on some form ofmachine-readable storage media, a ferromagnetic disk drive to storeprograms and/or data locally for a relatively extended period, and oneor more volatile solid state memory devices enabling relatively quickaccess to programs and/or data (e.g., SRAM or DRAM). It should also benoted that each of these memories may be made up of multiple storagecomponents based on identical storage technology, but which may bemaintained separately as a result of specialization in use (e.g., someDRAM devices employed as a main memory while other DRAM devices employedas a distinct frame buffer of a graphics controller).

In various embodiments, one or more of the interfaces 490, 590, 690,2490, 2590 and 2690 may employ any of a wide variety of signalingtechnologies enabling these computing devices to be coupled to otherdevices as has been described. Each of these interfaces includescircuitry providing at least some of the requisite functionality toenable such coupling. However, each of these interfaces may also be atleast partially implemented with sequences of instructions executed bycorresponding ones of the processor components (e.g., to implement aprotocol stack or other features). Where electrically and/or opticallyconductive cabling is employed, these interfaces may employ signalingand/or protocols conforming to any of a variety of industry standards,including without limitation, RS-232C, RS-422, USB, Ethernet(IEEE-802.3) or IEEE-1394. Where the use of wireless signal transmissionis entailed, these interfaces may employ signaling and/or protocolsconforming to any of a variety of industry standards, including withoutlimitation, IEEE 802.11a, 802.11b, 802.11g, 802.16, 802.20 (commonlyreferred to as “Mobile Broadband Wireless Access”); Bluetooth; ZigBee;or a cellular radiotelephone service such as GSM with General PacketRadio Service (GSM/GPRS), CDMA/1×RTT, Enhanced Data Rates for GlobalEvolution (EDGE), Evolution Data Only/Optimized (EV-DO), Evolution ForData and Voice (EV-DV), High Speed Downlink Packet Access (HSDPA), HighSpeed Uplink Packet Access (HSUPA), 4G LTE, etc.

FIG. 12 illustrates an embodiment of an exemplary processingarchitecture 3000 suitable for implementing various embodiments aspreviously described. More specifically, the processing architecture3000 (or variants thereof) may be implemented as part of one or more ofthe client devices 100, the administration devices 200, the nodes 300,the management modules 400, the network modules 500, the data modules600, the authoring devices 2100, the administration devices 2200, theingest server 2400, the database server 2500, the visualization server2600, the documentation server 2800, and the sets of storage devices 800ab, 800 cd or 800 yz. It should be noted that components of theprocessing architecture 3000 are given reference numbers in which thelast two digits correspond to the last two digits of reference numbersof at least some of the components earlier depicted and described aspart of the devices 100, 200, 800, 2100 and/or 2200; the servers 2400,2500, 2600 and/or 2800; and/or the modules 400, 500 and 600. This isdone as an aid to correlating components of each.

The processing architecture 3000 includes various elements commonlyemployed in digital processing, including without limitation, one ormore processors, multi-core processors, co-processors, memory units,chipsets, controllers, peripherals, interfaces, oscillators, timingdevices, video cards, audio cards, multimedia input/output (I/O)components, power supplies, etc. As used in this application, the terms“system” and “component” are intended to refer to an entity of acomputing device in which digital processing is carried out, that entitybeing hardware, a combination of hardware and software, software, orsoftware in execution, examples of which are provided by this depictedexemplary processing architecture. For example, a component can be, butis not limited to being, a process running on a processor component, theprocessor component itself, a storage device (e.g., a hard disk drive,multiple storage drives in an array, etc.) that may employ an opticaland/or magnetic storage medium, a software object, an executablesequence of instructions, a thread of execution, a program, and/or anentire computing device (e.g., an entire computer). By way ofillustration, both an application running on a server and the server canbe a component. One or more components can reside within a processand/or thread of execution, and a component can be localized on onecomputing device and/or distributed between two or more computingdevices. Further, components may be communicatively coupled to eachother by various types of communications media to coordinate operations.The coordination may involve the uni-directional or bi-directionalexchange of information. For instance, the components may communicateinformation in the form of signals communicated over the communicationsmedia. The information can be implemented as signals allocated to one ormore signal lines. A message (including a command, status, address ordata message) may be one of such signals or may be a plurality of suchsignals, and may be transmitted either serially or substantially inparallel through any of a variety of connections and/or interfaces.

As depicted, in implementing the processing architecture 3000, acomputing device includes at least a processor component 950, aninternal storage 960, an interface 990 to other devices, and a coupling959. As will be explained, depending on various aspects of a computingdevice implementing the processing architecture 3000, including itsintended use and/or conditions of use, such a computing device mayfurther include additional components, such as without limitation, adisplay interface 985.

The coupling 959 includes one or more buses, point-to-pointinterconnects, transceivers, buffers, crosspoint switches, and/or otherconductors and/or logic that communicatively couples at least theprocessor component 950 to the internal storage 960. Coupling 959 mayfurther couple the processor component 950 to one or more of theinterface 990 and the display interface 985 (depending on which of theseand/or other components are also present). With the processor component950 being so coupled by couplings 959, the processor component 950 isable to perform the various ones of the tasks described at length,above, for whichever one(s) of the aforedescribed computing devicesimplement the processing architecture 3000. Coupling 959 may beimplemented with any of a variety of technologies or combinations oftechnologies by which signals are optically and/or electricallyconveyed. Further, at least portions of couplings 959 may employ timingsand/or protocols conforming to any of a wide variety of industrystandards, including without limitation, Accelerated Graphics Port(AGP), CardBus, Extended Industry Standard Architecture (E-ISA), MicroChannel Architecture (MCA), NuBus, Peripheral Component Interconnect(Extended) (PCI-X), PCI Express (PCI-E), Personal Computer Memory CardInternational Association (PCMCIA) bus, HyperTransport™, QuickPath, andthe like.

As previously discussed, the processor component 950 may include any ofa wide variety of commercially available processors, employing any of awide variety of technologies and implemented with one or more coresphysically combined in any of a number of ways.

As previously discussed, the internal storage 960 may be made up of oneor more distinct storage devices based on any of a wide variety oftechnologies or combinations of technologies. More specifically, asdepicted, the internal storage 960 may include one or more of a volatilestorage 961 (e.g., solid state storage based on one or more forms of RAMtechnology), a non-volatile storage 962 (e.g., solid state,ferromagnetic or other storage not requiring a constant provision ofelectric power to preserve their contents), and a removable mediastorage 963 (e.g., removable disc or solid state memory card storage bywhich information may be conveyed between computing devices). Thisdepiction of the internal storage 960 as possibly including multipledistinct types of storage is in recognition of the commonplace use ofmore than one type of storage device in computing devices in which onetype provides relatively rapid reading and writing capabilities enablingmore rapid manipulation of data by the processor component 950 (butpossibly using a “volatile” technology constantly requiring electricpower) while another type provides relatively high density ofnon-volatile storage (but likely provides relatively slow reading andwriting capabilities).

Given the often different characteristics of different storage devicesemploying different technologies, it is also commonplace for suchdifferent storage devices to be coupled to other portions of a computingdevice through different storage controllers coupled to their differingstorage devices through different interfaces. By way of example, wherethe volatile storage 961 is present and is based on RAM technology, thevolatile storage 961 may be communicatively coupled to coupling 959through a storage controller 965 a providing an appropriate interface tothe volatile storage 961 that perhaps employs row and column addressing,and where the storage controller 965 a may perform row refreshing and/orother maintenance tasks to aid in preserving information stored withinthe volatile storage 961. By way of another example, where thenon-volatile storage 962 is present and includes one or moreferromagnetic and/or solid-state disk drives, the non-volatile storage962 may be communicatively coupled to coupling 959 through a storagecontroller 965 b providing an appropriate interface to the non-volatilestorage 962 that perhaps employs addressing of blocks of informationand/or of cylinders and sectors. By way of still another example, wherethe removable media storage 963 is present and includes one or moreoptical and/or solid-state disk drives employing one or more pieces ofmachine-readable storage medium 969, the removable media storage 963 maybe communicatively coupled to coupling 959 through a storage controller965 c providing an appropriate interface to the removable media storage963 that perhaps employs addressing of blocks of information, and wherethe storage controller 965 c may coordinate read, erase and writeoperations in a manner specific to extending the lifespan of themachine-readable storage medium 969.

One or the other of the volatile storage 961 or the non-volatile storage962 may include an article of manufacture in the form of amachine-readable storage media on which a routine including a sequenceof instructions executable by the processor component 950 may be stored,depending on the technologies on which each is based. By way of example,where the non-volatile storage 962 includes ferromagnetic-based diskdrives (e.g., so-called “hard drives”), each such disk drive typicallyemploys one or more rotating platters on which a coating of magneticallyresponsive particles is deposited and magnetically oriented in variouspatterns to store information, such as a sequence of instructions, in amanner akin to storage medium such as a floppy diskette. By way ofanother example, the non-volatile storage 962 may be made up of banks ofsolid-state storage devices to store information, such as sequences ofinstructions, in a manner akin to a compact flash card. Again, it iscommonplace to employ differing types of storage devices in a computingdevice at different times to store executable routines and/or data.

Thus, a routine including a sequence of instructions to be executed bythe processor component 950 may initially be stored on themachine-readable storage medium 969, and the removable media storage 963may be subsequently employed in copying that routine to the non-volatilestorage 962 for long-term storage not requiring the continuing presenceof the machine-readable storage medium 969 and/or the volatile storage961 to enable more rapid access by the processor component 950 as thatroutine is executed.

As previously discussed, the interface 990 may employ any of a varietyof signaling technologies corresponding to any of a variety ofcommunications technologies that may be employed to communicativelycouple a computing device to one or more other devices. Again, one orboth of various forms of wired or wireless signaling may be employed toenable the processor component 950 to interact with input/output devices(e.g., the depicted example keyboard 920 or printer 925) and/or othercomputing devices, possibly through a network (e.g., the network 999) oran interconnected set of networks. In recognition of the often greatlydifferent character of multiple types of signaling and/or protocols thatmust often be supported by any one computing device, the interface 990is depicted as including multiple different interface controllers 995 a,995 b and 995 c. The interface controller 995 a may employ any of avariety of types of wired digital serial interface or radio frequencywireless interface to receive serially transmitted messages from userinput devices, such as the depicted keyboard 920. The interfacecontroller 995 b may employ any of a variety of cabling-based orwireless signaling, timings and/or protocols to access other computingdevices through the depicted network 999 (perhaps a network made up ofone or more links, smaller networks, or perhaps the Internet). Theinterface controller 995 c may employ any of a variety of electricallyconductive cabling enabling the use of either serial or parallel signaltransmission to convey data to the depicted printer 925. Other examplesof devices that may be communicatively coupled through one or moreinterface controllers of the interface 990 include, without limitation,a microphone to monitor sounds of persons to accept commands and/or datasignaled by those persons via voice or other sounds they may make,remote controls, stylus pens, card readers, finger print readers,virtual reality interaction gloves, graphical input tablets, joysticks,other keyboards, retina scanners, the touch input component of touchscreens, trackballs, various sensors, a camera or camera array tomonitor movement of persons to accept commands and/or data signaled bythose persons via gestures and/or facial expressions, laser printers,inkjet printers, mechanical robots, milling machines, etc.

Where a computing device is communicatively coupled to (or perhaps,actually incorporates) a display (e.g., the depicted example display980), such a computing device implementing the processing architecture3000 may also include the display interface 985. Although moregeneralized types of interface may be employed in communicativelycoupling to a display, the somewhat specialized additional processingoften required in visually displaying various forms of content on adisplay, as well as the somewhat specialized nature of the cabling-basedinterfaces used, often makes the provision of a distinct displayinterface desirable. Wired and/or wireless signaling technologies thatmay be employed by the display interface 985 in a communicative couplingof the display 980 may make use of signaling and/or protocols thatconform to any of a variety of industry standards, including withoutlimitation, any of a variety of analog video interfaces, Digital VideoInterface (DVI), DisplayPort, etc.

More generally, the various elements of the computing devices describedand depicted herein may include various hardware elements, softwareelements, or a combination of both. Examples of hardware elements mayinclude devices, logic devices, components, processors, microprocessors,circuits, processor components, circuit elements (e.g., transistors,resistors, capacitors, inductors, and so forth), integrated circuits,application specific integrated circuits (ASIC), programmable logicdevices (PLD), digital signal processors (DSP), field programmable gatearray (FPGA), memory units, logic gates, registers, semiconductordevice, chips, microchips, chip sets, and so forth. Examples of softwareelements may include software components, programs, applications,computer programs, application programs, system programs, softwaredevelopment programs, machine programs, operating system software,middleware, firmware, software modules, routines, subroutines,functions, methods, procedures, software interfaces, application programinterfaces (API), instruction sets, computing code, computer code, codesegments, computer code segments, words, values, symbols, or anycombination thereof. However, determining whether an embodiment isimplemented using hardware elements and/or software elements may vary inaccordance with any number of factors, such as desired computationalrate, power levels, heat tolerances, processing cycle budget, input datarates, output data rates, memory resources, data bus speeds and otherdesign or performance constraints, as desired for a givenimplementation.

Some embodiments may be described using the expression “one embodiment”or “an embodiment” along with their derivatives. These terms mean that aparticular feature, structure, or characteristic described in connectionwith the embodiment is included in at least one embodiment. Theappearances of the phrase “in one embodiment” in various places in thespecification are not necessarily all referring to the same embodiment.Further, some embodiments may be described using the expression“coupled” and “connected” along with their derivatives. These terms arenot necessarily intended as synonyms for each other. For example, someembodiments may be described using the terms “connected” and/or“coupled” to indicate that two or more elements are in direct physicalor electrical contact with each other. The term “coupled,” however, mayalso mean that two or more elements are not in direct contact with eachother, but yet still co-operate or interact with each other.Furthermore, aspects or elements from different embodiments may becombined.

It is emphasized that the Abstract of the Disclosure is provided toallow a reader to quickly ascertain the nature of the technicaldisclosure. It is submitted with the understanding that it will not beused to interpret or limit the scope or meaning of the claims. Inaddition, in the foregoing Detailed Description, it can be seen thatvarious features are grouped together in a single embodiment for thepurpose of streamlining the disclosure. This method of disclosure is notto be interpreted as reflecting an intention that the claimedembodiments require more features than are expressly recited in eachclaim. Rather, as the following claims reflect, inventive subject matterlies in less than all features of a single disclosed embodiment. Thusthe following claims are hereby incorporated into the DetailedDescription, with each claim standing on its own as a separateembodiment. In the appended claims, the terms “including” and “in which”are used as the plain-English equivalents of the respective terms“comprising” and “wherein,” respectively. Moreover, the terms “first,”“second,” “third,” and so forth, are used merely as labels, and are notintended to impose numerical requirements on their objects.

1. An apparatus comprising: a processor component; a rendering componentto generate a visualization of at least a portion of a storage clustersystem for presentation on a display, the visualization to comprise adepiction of an object that corresponds to a component of the storagecluster system; and an interpretation component to interpret receivedindications of operation of an input device to select the depictedobject and to select a first time and a second time along a timelinepresented on the display, and to generate a command to requestinformation indicating a change in state of the object between the firstand second times.
 2. The apparatus of claim 1, comprising a userinterface (UI) component to operate the display and the input device toprovide a user interface to enable operation of the input device toselect the object and the first and second times, and to monitor theinput device to receive the indications of operation of the input deviceto select the object and the first and second times.
 3. The apparatus ofclaim 2, the UI component to present a menu of selectable menu items onthe display, each menu item to correspond to a different storage clustersystem, the UI component to monitor the input device to receive anindication of operation of the input device to select one of the menuitem that corresponds to the storage cluster system, and theinterpretation component to generate the command to specify the storagecluster system in response to the selection of the menu item.
 4. Theapparatus of claim 2, comprising: a network interface to couple theprocessor component to a network; and a communications component totransmit the command to a visualization server via the network interfaceto be translated into query instructions to search for a system entrycorresponding to the storage cluster system within a system database andinto query instructions to search for the requested information withinthe system entry.
 5. The apparatus of claim 4, the communicationscomponent to receive the requested information from the visualizationserver via the network interface, the rendering component to generate anew visualization based on the requested information to comprise adepiction of the change in state of the component between the first andsecond times as a change in state of the corresponding object betweenthe first and second times, the rendering component to generate a newtimeline to comprise at least one indication of an event that isassociated with the change in state of the object between the first andsecond times, and the rendering component to present the newvisualization and the new timeline on the display.
 6. The apparatus ofclaim 1, comprising a translation component of a visualization server totranslate the command to into query instructions to search for a systementry corresponding to the storage cluster system within a systemdatabase and into query instructions to search for the requestedinformation within the system entry.
 7. The apparatus of claim 6, therendering component to generate a new visualization based on therequested information to comprise a depiction of the change in state ofthe component between the first and second times as a change in state ofthe corresponding object between the first and second times, therendering component to generate a new timeline to comprise at least oneindication of an event that is associated with the change in state ofthe object between the first and second times, and the apparatuscomprising: a network interface of the visualization server to couplethe processor component to a network; and a communications component toreceive the command from another device via the network interface and totransmit a representation of the visualization to the other device viathe network interface.
 8. The apparatus of claim 6, comprising thesystem database and a database component to generate the system entry toinclude a separate object entry corresponding to each component ofmultiple components of the storage cluster system to treat eachcomponent of the multiple components as an object, and each object entryto comprise a data structure to store indications of various aspects ofthe operation of a corresponding one of the multiple components asproperties of the corresponding object.
 9. A computer-implemented methodcomprising: generating a visualization of at least a portion of astorage cluster system for presentation on a display, the visualizationto comprise a depiction of an object that corresponds to a component ofthe storage cluster system; generating a timeline for presentation onthe display; receiving indications of operation of an input deviceassociated with the display to select the depicted object and to selecta first time and a second time along the timeline; and generating acommand to request information indicating a change in state of theobject between the first and second times.
 10. The computer-implementedmethod of claim 9, comprising: operating the display and the inputdevice to provide a user interface (UI); presenting the visualizationand the timeline; and monitoring the input device to receive theindications of operation of the input device to select the object andthe first and second times.
 11. The computer-implemented method of claim10, comprising: presenting a menu of selectable menu items on thedisplay, each menu item to correspond to a different storage clustersystem; monitoring the input device to receive an indication ofoperation of the input device to select one of the menu item thatcorresponds to the storage cluster system; and generating the command tospecify the storage cluster system in response to the selection of theselectable menu item.
 12. The computer-implemented method of claim 10,comprising: transmitting the command to a visualization server via anetwork to be translated into query instructions to search for a systementry corresponding to the storage cluster system within a systemdatabase and into query instructions to search for the requestedinformation within the system entry; receiving the requested informationfrom the visualization server via the network; generating a newvisualization based on the requested information to comprise a depictionof the change in state of the component between the first and secondtimes as a change in state of the corresponding object between the firstand second times; generating a new timeline to comprise at least oneindication of an event that is associated with the change in state ofthe object between the first and second times; and presenting the newvisualization and the new timeline on the display.
 13. Thecomputer-implemented method of claim 9, comprising: receiving thecommand from another device via a network; translating the command intoquery instructions to search for a system entry corresponding to thestorage cluster system within a system database and into queryinstructions to search for the requested information within the systementry; generating a new visualization based on the requested informationto comprise a depiction of the change in state of the component betweenthe first and second times as a change in state of the correspondingobject between the first and second times; generating a new timeline tocomprise at least one indication of an event that is associated with thechange in state of the object between the first and second times; andtransmitting a representation of the visualization to the other devicevia the network.
 14. The computer-implemented method of claim 13,comprising: generating the system entry to include a separate objectentry corresponding to each component of multiple components of thestorage cluster system to treat each component of the multiplecomponents as an object, and each object entry to comprise a datastructure to store indications of various aspects of the operation of acorresponding one of the multiple components as properties of thecorresponding object; and in response to receiving the command,searching for the system entry within the system database and searchingfor the requested information within the system entry.
 15. At least onenon-transitory machine-readable storage medium comprising instructionsthat when executed by a processor component, cause the processorcomponent to: generate a visualization of at least a portion of astorage cluster system for presentation on a display, the visualizationto comprise a depiction of an object that corresponds to a component ofthe storage cluster system; generate a timeline for presentation on thedisplay; receive indications of operation of an input device associatedwith the display to select the depicted object and to select a firsttime and a second time along the timeline; and generate a command torequest information indicating a change in state of the object betweenthe first and second times.
 16. The at least one non-transitorymachine-readable storage medium of claim 15, the processor componentcaused to: operate the display and the input device to provide a userinterface (UI); present the visualization and the timeline; and monitorthe input device to receive the indications of operation of the inputdevice to select the object and the first and second times.
 17. The atleast one non-transitory machine-readable storage medium of claim 16,the processor component caused to: transmit the command to avisualization server via a network to be translated into queryinstructions to search for a system entry corresponding to the storagecluster system within a system database and into query instructions tosearch for the requested information within the system entry; receivethe requested information from the visualization server via the network;generate a new visualization based on the requested information tocomprise a depiction of the change in state of the component between thefirst and second times as a change in state of the corresponding objectbetween the first and second times; generate a new timeline to compriseat least one indication of an event that is associated with the changein state of the object between the first and second times; and presentthe new visualization and the new timeline on the display.
 18. The atleast one non-transitory machine-readable storage medium of claim 17,the processor component caused to: receive indications of operation ofthe input device to select one of the first and second times and a thirdtime along the timeline; and generate an altered visualization based onthe requested information to comprise a depiction of a change in stateof the component between the one of the first and second times and thethird time as a change in state of the corresponding object between theone of the first and second times and the third time.
 19. The at leastone non-transitory machine-readable storage medium of claim 15, theprocessor component caused to: receive the command from another devicevia a network; translate the command into query instructions to searchfor a system entry corresponding to the storage cluster system within asystem database and into query instructions to search for the requestedinformation within the system entry; generate a new visualization basedon the requested information to comprise a depiction of the change instate of the component between the first and second times as a change instate of the corresponding object between the first and second times;generate a new timeline to comprise at least one indication of an eventthat is associated with the change in state of the object between thefirst and second times; and transmit a representation of thevisualization to the other device via the network.
 20. The at least onenon-transitory machine-readable storage medium of claim 19, theprocessor component caused to: generate the system entry to include aseparate object entry corresponding to each component of multiplecomponents of the storage cluster system to treat each component of themultiple components as an object, and each object entry to comprise adata structure to store indications of various aspects of the operationof a corresponding one of the multiple components as properties of thecorresponding object; and in response to receiving the command, searchfor the system entry within the system database and search for therequested information within the system entry.